ASTROstream: Automated claSsification of Transient astRonomical phenOmena in the streaming mode

We present a real-time light curve classification system for the Zwicky Transient Facility (and the upcoming Large Synoptic Survey Telescope) alert streams based-on a Lambda Architecture (LA) and using Apache Kafka and Apache Spark. LA is a scalable and fault-tolerant data processing architecture that is designed to handle both real-time and historically aggregated batched data in an integrated fashion. Spark is a cluster computing framework which is widely used as an industry tool for processing and analyzing large data sets. We demonstrate how batches of data can be ingested and cross-matched against data from the PS1, SDSS, and other catalogs, and how the light curves from these data can be rapidly characterized by scaling existing Python applications to large data volumes. We produce a catalog of the transient sky, including a large and diverse range of phenomena: variable events, periodic …