4. Ordering Metrics

A crawler keeps a queue of URLs it has seen during the crawl, and must select from this queue the next
URL to visit. The ordering metric O is used by the crawler for this selection, i.e., it selects the URL u
such that O(u) has the highest value among all URLs in the queue. The O metric can only use
information seen (and remembered if space is limited) by the crawler.

The O metric should be designed with an importance metric in mind. For instance, if we are searching
for high IB(P) pages, it makes sense to use an O(u) = IB’(P), where P is the page u points to. However,
it might also make sense to consider an O(u) = IR’(P), even if our importance metric is not weighted. In
our experiments, we explore the types of ordering metrics that are best suited for either IB(P) or IR(P).

For a location IL(P) importance metric, we can use that metric directly for ordering since the URL of P
directly gives the IL(P) value. However, for forward link IF(P) and similarity IS(P) metrics, it is much
harder to devise an ordering metric since we have not seen P yet. As we will see, for similarity, we may
be able to use the text that anchors the URL u as a predictor of the text that P might contain. Thus, one
possible ordering metric O(u) is IS(A, Q), where A is the anchor text of the URL u, and Q is the driving
query.