You saw that they found SARS using their Chinese web crawler a full six weeks before they found it using their English web crawler.

Of course you can. You saw that they found SARS using their Chinese web crawler a full six weeks before they found it using their English web crawler.

A crawler must carefully choose at each step which pages to visit next.

It then sorts these pages into a big database inside the crawler and also analyzes developments of each page to any possible query.

As a crawler always downloads just a fraction of the Web pages, it is highly desirable that the downloaded fraction contains the most relevant pages and not just a random sample of the Web.

When the Web came out, the early Web crawlers were called robots and to block a robot from accessing your website, to the present day, there's a file called robot.txt, that allows you to deny any Web crawler to access and retrieve that information from your website.