Dealing with web data: history and look ahead

The high rate of change and the unprecedented scale of the Web pose enormous challenges to search engines who wish to provide the most up-to-date and highly relevant information to its users. The VLDB 2000 paper "The Evolution of the Web and Implications for an Incremental Crawler" tried to address part of this challenge by collecting and analyzing the Web history data and by describing the architecture and the associated algorithms for an incremental Web crawler that can provide more up-to-date data to users in a timely manner. Experiments and theoretical analysis showed --- surprisingly at the time --- that a policy that allocates more resources to more frequently changing items does not necessarily lead to better performance. In this paper, we discuss what has happened in the 10 years since and talk about the challenges that lie head.