Crawling AJAX by Inferring User Interface State Changes

Ajax is a very promising approach for improving rich interactivity and responsiveness of web applications. At the same time, Ajax techniques shatter the metaphor of a web "page" upon which general search crawlers are based. This paper describes a novel technique for crawling Ajax applications through dynamic analysis and reconstruction of user interface state changes. Our method dynamically infers a state-flow graph modeling the various navigation paths and states within an Ajax application. This reconstructed model can be used to generate linked static pages. These pages could be used to expose Ajax sites to general search engines. Moreover, we believe that the crawling techniques that are part of our solution have other applications, such as within general search engines, accessibility improvements, or in automatically exercising all user interface elements and conducting state-based testing of Ajax applications. We present our open source tool called Crawljax which implements the concepts discussed in this paper. Additionally, we report a case study in which we apply our approach to a number of representative Ajax applications and elaborate on the obtained results.