A Query based Approach to Reduce the Web Crawler Traffic using HTTP Get Request and Dynamic Web Page

The functions of Web crawler download information from web for search engine. Web pages changed without any notice. Web crawler has to revisit web site to download updated and new web pages. It is estimated 40% of current web traffic is generated by web crawler. This paper proposes query based approach to inform updates on web site to web crawler using Dynamic web page and HTTP GET Request. Dynamic web page generates HTML based response having list of updates on web site after crawler last visit. Web crawler only visits updated web pages instead of visiting full web sites for updates. Proposed scheme is tested & results show that it is very promising.