In this paper we addressed the problem of ordering URLs for crawling. We listed different kinds of
importance metrics, and built three models to evaluate crawlers. We experimentally evaluated several
combinations of importance and ordering metrics, using the Stanford Web pages.
In general our results show that PageRank, IR’(P), is an excellent ordering metric when either pages
with many backlinks or with high PageRank are sought. In addition, if the similarity to a driving query is
important, then it is also useful to visit earlier URLs that:
Have anchor text that is similar to the driving query;
Have some of the query terms within the URL itself; or
Have a short link distance to a page that is known to be hot.
With a good ordering strategy, it seems to be possible to build crawlers that can rather quickly obtain a
significant portion of the hot pages. This can be extremely useful when we are trying to crawl large
portions of the Web, when are resources are limited, or when we need to revisit pages often to detect
changes.
No comments:
Post a Comment