Abstract
In this paper we study in what order a crawler should visit the URLs it has seen, in order to
obtain more "important" pages first. Obtaining important pages rapidly can be very useful
when a crawler cannot visit the entire Web in a reasonable amount of time. We define
several importance metrics, ordering schemes, and performance evaluation measures for
this problem. We also experimentally evaluate the ordering schemes on the Stanford
University Web. Our results show that a crawler with a good ordering scheme can obtain
important pages significantly faster than one without.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment