5. Experimental Setup

To avoid network congestion and heavy loads on the servers, we did our experimental evaluation in two
steps. In the first step, we physically crawled all Stanford Web pages and built a local repository of the
pages. This was done with the Stanford WebBase, a system designed to create and maintain large web
repositories. It is capable of high indexing speeds (about 50 pages per second), and large data
repositories (currently 150GB of HTML is stored).

After we built the repository, we ran our virtual crawlers on it to evaluate the different crawling
schemes. Note that even though we had the complete image of the Stanford domain in the repository,
our virtual crawler based its crawling decisions only on the pages it saw for itself. In this section we
briefly discuss how the WebBase crawler operates, and how the particular database was obtained for our
experiments.

5.1 WebBase Crawler

WebBase runs several processes at a time, which crawl web pages. These processes receive a list of
URLs to be downloaded and simply return the full content of the HTML or any errors which happened
trying to get the pages. The crawling processes open several hundred connections at a time, resulting in
a crawling speed of about 25 pages/second for each process. Since servers can only handle a few tens of
hits per second at most, and slowing down servers with a web crawler is a problem, we use two different
kinds of load balancing in our system. First, our system splits all URLs which are going to be crawled
into 500 queues based on a hash of their server name. This causes all URLs from a given server to go
into the same queue. The crawlers then read one URL from each queue at a time, moving to a new queue
for each URL. This makes sure a given server is hit only once for every 500 URLs that are crawled.
Also, for servers that are slow at returning documents, only one connection is allowed from the crawler
to a particular server at a time. As is mentioned in Section 1, these kinds of load balancing requirements
affect the freedom a crawler has in deciding the crawl order.

The actual data the system is allowed to get is reduced for two reasons. The first is that many heuristics
are needed to avoid automatically generated, and potentially infinite, sets of pages. For example, any
URLs containing "/cgi-bin/" are not crawled, because they are likely to contain programs which generate
infinite sets of pages, or producing other undesirable side effects such as an unintended vote in an online
election. Several other heuristics based on the Location Metric described above are used to weed out
URLs which look undesirable. Another way the data set is reduced is through the robots exclusion
protocol[7], which allows webmasters to define pages they do not want crawled by automatic systems.

5.2 Description of Dataset

To download an image of the Stanford web pages, we started WebBase with an initial list of
"stanford.edu" URLs. These 89,119 URLs were obtained from an earlier crawl. During the crawl,
non-Stanford URLs were ignored. At the end of the process, we had 784,592 known URLs to Stanford
pages. It should be noted that 352,944 of the known URLs were on one server, www.slac.stanford.edu
[8], which has a program that generates infinite numbers of web pages. The crawl was stopped before it
was complete, but most of the uncrawled URLs were on only a few servers so we believe the dataset we
used to be a reasonable representation of the stanford.edu web. This dataset consisted of about 225,000
crawled valid HTML pages using roughly 2.5 GB of space for the raw pages.

We should stress that the virtual crawlers that will be discussed next do not use WebBase directly. As

stated earlier, they use the dataset collected by the WebBase crawler, and do their own crawling on it.
The virtual crawlers are simpler than the Web Base crawler. For instance, they can detect if a URL is
invalid simply by seeing if it is in the dataset. Similarly, they do not need to distribute the load to visited
sites. These simplifications are fine, since the virtual crawlers are only used to evaluate ordering
schemes, and not to do real crawling.