Katti: An Extensive and Scalable Tool for Website Analyses.

Research on web security and privacy frequently relies on tools that analyze a set of websites. One major obstacle to the judicious analysis is the employment of a rock-solid and feature-rich web crawler. For example, the automated analysis of ad-malware campaigns on websites requests crawling a vast set of domains on multiple real web browsers, while simultaneously mitigating bot detections and applying user interactions on websites. Further, the ability to attach various threat analysis frameworks lacks current tooling efforts in web crawling and analyses. In this paper we introduce Katti, which overcomes several of today’s technical hurdles in web crawling. Our tool employs a distributed task queue that efficiently and reliably handles both large crawling and threat analyses requests. Katti  extensively collects all available web data through an integrated person-in-the-middle proxy. Moreover, Katti  is not limited to a specific use case, allowing users to easily customize our tool to their individual research intends.