Optimal Freshness Crawl Under Politeness Constraints
Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 495-504, 2019.
A Web crawler is an essential part of a search engine that procures information subsequently served by the search engine to its users. As the Web is becoming increasingly more dynamic, in addition to discovering new web pages a crawler needs to keep revisiting those already in the search engine's index, in order to keep the index fresh by...More
Full Text (Upload PDF)
PPT (Upload PPT)