Exploratory data analysis is the primary technique used by data scientists to extract knowledge from new data sets. This type of workload is composed of trial-and-error hypothesis-driven queries with a human in the loop. To keep up with the data scientist’s productivity, the system must be capable of answering queries in interactive times. Given that these queries are highly selective multidimensional queries, multidimensional indexes are necessary to ensure low latency. However, creating the appropriate indexes is not a given due to the highly exploratory and interactive nature of such human-in-the-loop scenarios.In this paper, we identify four main objectives that are desirable for exploratory data analysis workloads: (1) low overhead over the initial queries, (2) low query variance (i.e., high robustness), (3) predictable index convergence, and (4) low total workload time. Given that not all of them can be achieved at the same time, we present three novel incremental multidimensional indexing techniques that represent three sample points on a Pareto front for this multi-objective optimization problem. (a) The Adaptive KD-Tree is designed to achieve the lowest total workload time at the expense of a higher indexing penalty for the initial queries, lack of robustness, and unpredictable convergence. (b) The Progressive KD-Tree has predictable convergence and a user-defined indexing cost for the initial queries. However, total workload time can be higher than with Adaptive KD-Trees, and per-query time still varies. (c) The Greedy Progressive KD-Tree aims at full robustness at the expense of only improving the per-query cost after full index convergence.Our extensive experimental evaluation using both synthetic and real-life data sets and workloads shows that (a) the Adaptive KD-Tree reduces total workload time by up to a factor 2 compared to the state-of-the-art, (b) the Progressive KD-Tree achieves predictable convergence with up to one order of magnitude lower initial query cost, and (c) the Greedy Progressive KDTree exhibits the lowest query variance up to three orders of magnitude lower than the state-of-the-art.

Many practical data management applications are subject to real-time constraints, i.e., upper limits on response time of each operation. In the past, database research has mostly focused on what is known as “soft” real-time applications, i.e., those that expect results in a timely manner but only suffer minor degradation if the constraints are violated. However, data management systems are becoming increasingly common in applications with “firm” or even “hard” real-time requirements. State-of-the-art cars, for example, are engineered as collections of semi-autonomous components that are connected using a real-time Bus such as CAN. The components (sensors, actuators, steering devices, ...) update a centralized database which forms the basis for hard real-time decisions (such as controlling the Anti-lock Brake System or activating the airbag) [2]. Hard real-time systems are difficult to implement on top of general purpose CPUs due to hardware design decisions that favor throughput over performance predictability: out-oforder execution, cache-line prefetching, pre-emptive multithreading and many more. Similarly, most high-performance transaction processors focus on throughput (i.e., average latency) rather than deterministic latency. This makes them inherently unsuited for hard real-time applications. To illustrate the difficulty of providing daterministic latencies in throughput-oriented transaction processing systems, we evaluate the YCSB write transactions on the state-of-theart transaction processor Cicada [1]. To minimize variance caused by contention, we limit Cicada to a single worker thread and present the histogram of response times in Figure 1. Note that there is almost two orders of magnitude difference between the average latency and that of the slowest transaction. While this is not problematic for throughputoriented systems, this violates the requirements of efficient hard real-time applications: consider an (arbitrary) constraint of 60 us (illustrated by the purple vertical line in operations that violate the time constraint count as ”failed” even one violated constraint is considered a system failure This article is published under a Creative Commons Attribution Licens (http://creativecommons.org/licenses/by/3.0/), which permits distribution and reproduction in any medium as well as allowing derivative works, provided that you attribute the original work to the author(s) and CIDR 2020. 10th Annual Conference on Innovative Data Systems Research (CIDR ’20). January 12-15, 2020, Amsterdam, Netherlands. CIDR ’20 January 12–15, 2020, Amsterdam, Netherlands c © 2019 ACM. ISBN 978-1-4503-2138-9. DOI: 10.1145/1235 Misprovisioned Resources Real-Time Constraint Constraint Violations Figure 1: Histogram of Cicada’s per query latency, executing 1 million YCSB writes (theta = 0.1) on 10 million rows. Figure 1). Any transaction to the right of this line is considered a failure while any transaction that finishes earlier than the constraint is an indication of misprovisioned resources, i.e., those that could be used to improve secondary objectives such as energy efficiency or throughput. As apparent from the figure, a constraint cannot be set such that it satisfies the real-time requirement and uses resources efficiently. We argue that efficient hard real-time transaction processing is not achievable in software but requires the use of customizable hardware such as Field-Programmable Gate Arrays (FPGAs), which have very low and predictable latency. In fact, FPGAs have been used in automotive applications for exactly that reason [2]. However, the inherently parallel nature of FPGAs makes them tricky to program correctly. To address that problem, we are developing an FPGAbased real-time transaction processor. While data mangement on FPGAs has received interest from researchers in the past, the focus was almost exclusively on data analytics. We conjecture that this is, to a large extent, due to persistence being an unsolved problem. However, the emergence of non-volatile memory is likely to change that which opens a window of opportunity for research in that space. While we concede that there are significant challenges in developing such a system (among them concurrency control, indexing, buffer management, scaling), we believe that now is the right time to study transaction processing on FPGAs and exploit their efficient hard real-time guarantees.

Workload-aware physical data access structures are crucial to achieve short response time with (exploratory) data analysis tasks as commonly required for Big Data and Data Science applications. Recently proposed techniques such as automatic index advisers (for a priori known static workloads) and query-driven adaptive incremental indexing (for a priori unknown dynamic workloads) form the state-of-the-art to build single-dimensional indexes for single-attribute query predicates. However, similar techniques for more demanding multi-attribute query predicates, which are vital for any data analysis task, have not been proposed, yet. In this paper, we present our on-going work on a new set of workload-adaptive indexing techniques that focus on creating multidimensional indexes. We present our proof-of-concept, the Cracking KD-Tree, an adaptive indexing approach that generates a KD-Tree based on multidimensional range query predicates. It works by incrementally creating partial multidimensional indexes as a by-product of query processing. The indexes are produced only on those parts of the data that are accessed, and their creation cost is effectively distributed across a stream of queries. Experimental results show that the Cracking KD-Tree is three times faster than creating a full KD-Tree, one order of magnitude faster than executing full scans and two orders of magnitude faster than using uni-dimensional full or adaptive indexes on multiple columns.