Abstract Complex event processing (CEP) is an essential technology for analyzing streams of events. A key feature of a modern CEP architecture is the ability to process both continuous queries and analytical ad hoc queries on high-volume streams. Both query types support common operations (filter, aggregation, joins) known in event stream and database systems. Additionally, a crucial and unique operation in CEP is pattern matching, which matches user-defined predicates to subsequences of events. We present our solution for a system supporting continuous queries, fast ingestion, and efficient analytical ad hoc queries. The system follows the principles of a Lambda Architecture and is specialized for a large variety of pattern-matching queries, including sequential, situation, and group patterns. To offer efficient processing, we use modern hardware in each of the components. For continuous queries, we explore multi-core CPUs and GPUs. For ingestion and ad hoc queries, we analyze SSDs and persistent memory as ways to provide a robust system. Furthermore, we explore unique characteristics of the hardware and event processing applications such as temporal data, energy efficiency, and compression. We give an overview of the overall systems, highlight the research accomplishments, and describe common application scenarios that benefit from our architecture.

Pattern matching, also known as Match-Recognize in SQL, is an expensive operator of particular relevance in many event stream applications. However, because of its sequential nature and challenging latency requirements, current stream processing engines do not provide any parallel processing support for pattern matching. In addition, hardware accelerators based on dedicated GPUs also offer limited support due to the overhead of transferring data between their local and main memory. In contrast, however, integrated GPUs (iGPUs), with their ability to access main memory directly, offer great potential to accelerate pattern matching. This paper presents the first full-fledged implementation of pattern matching cooperatively using iGPUs and CPUs. Our results obtained from a preliminary experimental performance comparison confirm the potential of our iGPU-based approaches for accelerating pattern matching.

IoT applications require a new type of database systems termed event stores for ingesting fast arriving event streams and efficiently supporting analytical ad-hoc queries over time. One of the most important operations in this regard is sequential pattern matching also known as Match\_Recognize, which matches user defined predicates to subsequences of events. While Match\_Recognize is well known in the field of event processing, it has only recently become part of the SQL standard. Despite of that, Match\_Recognize has received little attention in the database area so far. We present a novel approach to speed up an important class of Match\_Recognize queries on event stores by utilizing off-the-shelf secondary indexes on non-temporal attributes (e.g., B$^+$-trees, LSM-trees) and a cost model for selecting the most appropriate indexes. Our approach keeps temporal and sequential information in secondary indexes to prune large parts of the stream from further processing. However, simply using as many secondary indexes as available is not the right choice because the access cost for the index scans can exceed the processing time of the naï ve approach that scans the entire stream and replays it into an event processing system. In order to address this problem, we present a first cost model to estimate the total execution cost of a Match\_Recognize query for a set of available indexes. Based on this cost model, we devise an efficient index selection strategy that avoids a full enumeration of index configurations. Prototypical implementations of our approach are available in our open-source research prototype, a commercial database system, and Apache Flink. In experiments with synthetic and real-world data sets, all our index-based implementations clearly outperform the naï ve replay strategy that is currently offered in commercial database systems and Flink.

Event stores face the difficult challenge of continuously ingesting massive temporal data streams while satisfying demanding query and recovery requirements. Many of today’s systems deal with multiple hardware-based trade-offs. For instance, long-term storage solutions balance keeping data in cheap secondary media (SSDs, HDDs) and performance-oriented main-memory caches. As an alternative, in-memory systems focus on performance, while sacrificing monetary costs, and, to some degree, recovery guarantees. The advent of persistent memory (PMem) led to a multitude of novel research proposals aiming to alleviate those trade-offs in various fields. So far, however, there is no proposal for a PMem-powered specialized event store. Based on ChronicleDB, we will present several complementary approaches for a three-layer architecture featuring main memory, PMem, and secondary storage. We enhance some of ChronicleDB’s components with PMem for better insertion and query performance as well as better recovery guarantees. At the same time, the three-layer architecture aims to keep the overall dollar cost of a system low. The limitations and opportunities of a PMem-enhanced event store serve as important groundwork for comprehensive system design exploiting a modern storage hierarchy.

Reactive security monitoring, self-driving cars, the Internet of Things (IoT), and many other novel applications require systems for both writing events arriving at very high and fluctuating rates to persistent storage as well as supporting analytical ad hoc queries. As standard database systems are not capable of delivering the required write performance, log-based systems, key-value stores, and other write-optimized data stores have emerged recently. However, the drawbacks of these systems are a fair query performance and the lack of suitable instant recovery mechanisms in case of system failures. In this article, we present ChronicleDB, a novel database system with a storage layout tailored for high write performance under fluctuating data rates and powerful indexing capabilities to support a variety of queries. In addition, ChronicleDB offers low-cost fault tolerance and instant recovery within milliseconds. Unlike previous work, ChronicleDB is designed either as a serverless library to be tightly integrated in an application or as a standalone database server. Our results of an experimental evaluation with real and synthetic data reveal that ChronicleDB clearly outperforms competing systems with respect to both write and query performance.

Sequential pattern matching to detect a user-defined sequence of conditions on event streams is a key feature in modern event processing systems. However, the sequential nature of event based pattern matching has two major deficiencies. First, it is hardly possible to express complex temporal relationships between situations lasting for periods of time. Because events are equipped with a single timestamp only, the expressible temporal relations are limited to before/after/at the same time. Second, a sequential pattern is mapped to a continuous subsequence of the input stream starting with an arbitrary event, making efficient parallelization a hard problem. In this paper we present TPStream, a novel event processing operator for complex temporal pattern matching on event streams. TPStream first summarizes incoming events to situations lasting for periods of time, before it matches temporal patterns. With situations, temporal patterns can easily be defined based on Allen’s interval algebra. We also show that situation based temporal pattern matching can be efficiently executed in parallel using multiple threads on a single machine or multiple machines in a cluster. Finally, we present adaptive optimization components continuously tuning the execution strategy of TPStream towards the lowest possible result latency with respect to the overall system load. The results of our experimental evaluation show that TPStream is capable of processing high-volume event streams with both low latency and high throughput while outperforming applicable CEP solutions from academia and industry.

A key technology to analyze high volume spatio-temporal data streams is complex event processing (CEP). CEP is unique in its ability to not only continuously process data as it arrives through common operations such as aggregations, but also to support pattern matching queries. Pattern Matching allows to detect a user-defined sequence of temporal predicates on event streams. The high volume flight data as provided by the OpenSky Network has a lot of characteristics that make it a perfect match for CEP. In particular, pattern matching operators can be utilized to detect a plethora of movement (landing, starting, evasion) and group patterns (airplanes closing in on each other) in a timely manner. However, CEP queries can be complex in nature and may require a combination of domain expertise and historical data analysis in order to deliver the desired results. In order to address these issues, we have combined a database-backed CEP system (ChronicleDB) with a scientific toolbox for interactive data exploration and geo visualization (Vat System). This allows users to interactively execute CEP queries and visually confirm the validity of their results, thus, simplifying the parameter tuning considerably. In addition, our solution supports efficient and interactive time travel queries. It allows to combine event streams with additional data sources (e.g., remote sensing images) and processing technologies (e.g., machine learning models) to extract higher level knowledge. Finally, our ongoing work on visual analytics explores extrapolating query results to provide more timely feedback for critical situations and multi-query optimization techniques to allow for an even more efficient system in general.

Reactive security monitoring, self-driving cars, the Internet of Things (IoT), and many other novel applications require systems for both writing events arriving at very high and fluctuating rates to persistent storage as well as supporting analytical ad hoc queries. As standard database systems are not capable of delivering the required write performance, log-based systems, key-value stores, and other write-optimized data stores have emerged recently. However, the drawbacks of these systems are a fair query performance and the lack of suitable instant recovery mechanisms in case of system failures. In this article, we present ChronicleDB, a novel database system with a storage layout tailored for high write performance under fluctuating data rates and powerful indexing capabilities to support a variety of queries. In addition, ChronicleDB offers low-cost fault tolerance and instant recovery within milliseconds. Unlike previous work, ChronicleDB is designed either as a serverless library to be tightly integrated in an application or as a standalone database server. Our results of an experimental evaluation with real and synthetic data reveal that ChronicleDB clearly outperforms competing systems with respect to both write and query performance.

Due to the widespread availability of general purpose GPUs, an integration of their processing capabilities into an event stream pipeline presents an exciting opportunity riddled with challenging requirements: Even though the single instruction multiple data (SIMD) model is a natural fit to answer long running event queries on high volume streams, those queries are usually associated with latency requirements that make transferring data to GPUs unfeasible. Traditionally, this challenge is solved through software by scheduling some tasks to the GPU and some to the CPU. However, the assumptions about transfer do not hold for widely adopted integrated GPUs (iGPUs), which directly share memory with the CPU. We develop a prototypical event processing framework based on the Heterogeneous System Architecture (HSA) and show that a variety of new HSA features enable iGPUs to be an affordable accelerator for a wide variety of event processing queries.

Mobile devices are increasingly being used in edge and fog computing environments to process contextual data collected by sensors. Although complex event processing (CEP) is a suitable approach for realizing context-aware services on mobile devices in these environments, existing mobile CEP engines do not leverage the full potential of modern mobile hardware/software architectures. In this paper, we present multimodal CEP, a novel approach to process streams of events on-device in user space (user mode), in the operating system (kernel mode), on the Wi-Fi chip (Wi-Fi mode), and/or on a sensor hub (hub mode), providing significant improvements in terms of power consumption and throughput. Multimodal CEP automatically breaks up CEP queries and selects the most adequate execution mode for the involved CEP operators. Filter, aggregation, and correlation operators can be expressed in a high-level language without requiring system-level domain-specific knowledge. Multimodal CEP enables developers to efficiently detect user activities, collect environmental conditions, or interpret operating system and network events. Furthermore, it facilitates novel context-aware services, demonstrated by a use case for gathering and analyzing mobility data by Wi-Fi probe request tracking.

Complex Event Processing (CEP) has emerged as the state-ofthe-art technology for continuously monitoring and analyzing streams of events in time-critical applications. The key feature in CEP is sequential pattern matching to detect a user-defined sequence of conditions on event streams. However, many CEP applications are not restricted to events only, but require native support for situations (aggregated event data lasting periods of time) and expressive temporal pattern matching among these situations. These important requirements regarding situations are not sufficiently addressed in the CEP literature so far. In this paper we present TPStream, a novel event-processing operator for both deriving situations from event streams and detecting temporal patterns among situations. First, we provide a formal foundation of situations and TPStream. Then, we propose a low-latency algorithm for TPStream that delivers situations and temporal matches at the earliest possible point in time. Furthermore, we utilize a simple, yet effective costmodel in order to adapt to changing workloads on the fly and with negligible cost for migrating operator states. The results of our experimental evaluation show that TPStream is capable of processing high-volume event streams with low latency and outperforms applicable CEP solutions from academia and industry.