Twitter Sparrow - Reduce Event Pipeline latency from hours to seconds.

Data Analytics at Twitter rely on trillions of events generated daily by micro services backing the Twitter platform. Multiple features on Twitter Mobile Application and Web interface are backed by micro services which emit events triggered by user actions. Events are well defined structured objects with fields containing important information relevant to the feature it represents. These events are aggregated and processed before making it available for internal consumption at various storage systems. Event processing pipelines in the past were designed to support scale in the order of billions of events per minute [1]. Events flowing through various software systems were batched to optimize for throughput in favor of latency. These batched event pipelines observe an end to end latency of a few hours, from the time when the event is emitted to when it is ready to be consumed. Project Sparrow is redesigning this pipeline to reduce the event latency from hours to seconds (or minutes).