Index is an important component in database systems. Learned indices have been shown to outperform traditional tree-based index structures for fixed-sized integer or floating point keys. However, the application of the learned solution to variable-length string keys is under-researched. Our experiments show that existing learned indices for strings fail to outperform traditional string indices, such as HOT and ART. String keys are long and variable sized, and often contain skewed prefixes, which make the last-mile search expensive, and adversely impact the capability of learned models to capture the skewed distribution of string keys. In this paper, we propose a novel learned index for string keys, LITS ( L earned I ndex with Hash-enhanced Prefix T able and S ub-tries). We design an optimized learned model that combines a global Hash-enhanced Prefix Table (HPT) with per-node local linear models to better distinguish string keys. Moreover, we exploit two techniques, i.e., dynamic scale factor and cell-based root, to improve the space efficiency. We prove that the resulting design is guaranteed to have O ( N ) space cost. Furthermore, we leverage compact leaf nodes and hybrid structures with a PMSS model to support efficient point and range queries. Our experimental results using eleven string data sets show that LITS achieves up to 2.19 × and 1.85 × improvement over HOT and ART for point operations with comparable scan performance and space cost.
The incorporation of domain knowledge (DK) in AI has been studied for years and turns out to be critical in practice. Functions are also a basic notion for dealing with data science projects and are somehow related to DK. Consider the following scenario. Let \(D(y, z_1, \ldots , z_n)\) be a dataset, Alice a data scientist, Bob a domain expert and \(y = f(z_1, \ldots , z_n)\) a function known to Bob from his background knowledge. Alice is interested in the following simple yet crucial questions: How to define the satisfaction of f in D ? How to measure that satisfaction efficiently? How does this satisfaction relate to the supervised learning task of learning f from D ? It turns out that these problems are related to the study of counterexamples through the use of functional dependencies (FDs) and, in particular, FD measures used to quantify their satisfaction in a dataset such as the \(g_3\) indicator where the equality is replaced by more flexible predicates. In this article, we first examine the complexity of computing \(g_3\) . It is known that \(g_3\) can be computed in polynomial time when using equality, while it becomes NP -hard when using general predicates. Our goal is to refine this dichotomy by studying the impact of the following common properties: reflexivity, transitivity, symmetry, and antisymmetry. We show that symmetry and transitivity together are sufficient to guarantee that the \(g_3\) can be computed in polynomial time. However, removing one of them makes the problem NP -hard. Second, we study the computation of \(g_3\) in the polynomial and NP -hard cases identified previously. We propose different exact and approximate solutions for the computation of \(g_3\) in both cases. We compare these solutions in a detailed experimental study of time performance and approximation accuracy. All the algorithms are also made available via fastg3 , an open-source Python library with an underlying C++ implementation. Finally, we link counterexamples and \(g_3\) to supervised learning with a web application called adesit . adesit is intended to be part of an iterative data refinement process right after data selection and just before the machine learning process itself. It provides a way to evaluate the ability of a dataset to perform well for a given supervised learning problem through statistical and visual exploration. In a last section, we validate our approach by applying it to the industrial problem of air gap monitoring in compact hydro-generators.
A data structure is history independent if its internal representation reveals nothing about the history of operations beyond what can be determined from the current contents of the data structure. History independence is typically viewed as a security or privacy guarantee, with the intent being to minimize risks incurred by a security breach or audit. Despite widespread advances in history independence, there is an important data-structural primitive that previous work has been unable to replace with an equivalent history-independent alternative— dynamic partitioning . In dynamic partitioning, we are given a dynamic set S of ordered elements and a size-parameter B , and the objective is to maintain a partition of S into ordered groups, each of size Θ ( B ). Dynamic partitioning is important throughout computer science, with applications to B-tree rebalancing, write-optimized dictionaries, log-structured merge trees, other external-memory indexes, geometric and spatial data structures, cache-oblivious data structures, and order-maintenance data structures. The lack of a history-independent dynamic-partitioning primitive has meant that designers of history-independent data structures have had to resort to complex alternatives. In this paper, we achieve history-independent dynamic partitioning. Our algorithm runs asymptotically optimally against an oblivious adversary, processing each insert/delete with O (1) operations in expectation and O ( B log N /log logN ) with high probability in set size N . We also use our dynamic partitioning scheme to build a history-independent B -tree, history-independent fusion tree, and external-memory skip list.
We study the indexed set intersection problem. Given a family \(\mathcal {S}=\lbrace S_1,\ldots ,S_N\rbrace\) of sorted integer sets, preprocess \(\mathcal {S}\) to answer queries \(Q\subseteq [1{..}N]\) that ask to compute \(\bigcap _{i\in Q} S_i\) . We introduce trie certificates , a new approach for the adaptive analysis of set intersection algorithms. Trie certificates capture the difficulty of an instance and allow one to obtain adaptive bounds for the set intersection problem across multiple computational models. Using trie certificates, we revisit Trabb-Pardo’s algorithm [Trabb-Pardo, 1978] in the pointer-machine model and prove an adaptive worst-case bound that matches the optimal comparison-model adaptive bound of Barbay and Kenyon [Barbay and Kenyon, 2008], namely \(\mathrm{O}(\delta \sum _{i\in Q}\lg \bigl (\frac{n_i}{\delta }\bigr))\) , where \(|S_i| = n_i\) and \(\delta\) is the alternation measure of Q . We also introduce a stronger difficulty measure \(\xi \le \delta\) , called the run alternation measure, and design an algorithm whose running time adapts to \(\xi\) , achieving \(\mathrm{O}(\xi \sum _{i\in Q}\lg \bigl (\frac{n_i}{\xi }\bigr))\) time on a pointer machine. On a transdichotomous word-RAM with word size w , we show that these algorithms can be implemented with a \(\Theta (w)\) -factor reduction in space, while retaining the above running times. Also on a transdichotomous word-RAM, via a decomposition of the original intersection instance into smaller subinstances, we show that the running time can be further improved. We then extend trie certificates to the thread-parallel setting by also decomposing each query into independent subinstances over disjoint subuniverses. We obtain a running-time bound, that is, determined by the hardest subinstance, and highlight the challenge of balancing difficulty across threads. Our experimental results empirically support our main findings, showing that our algorithms are competitive in practice and provide attractive space–time trade-offs.
JSON Schema is an important, evolving standard schema language for families of JSON documents. It is based on a complex combination of structural and Boolean operators, including negation, as well as mutually recursive variables. The static analysis of JSON Schema documents comprises practically relevant problems, including schema satisfiability, inclusion, and equivalence. These three can be reduced to witness generation: given a schema, generate an element of the schema — if it exists — otherwise report unsatisfiability. Schema satisfiability, inclusion, and equivalence have been shown to be decidable, by reduction to reachability in alternating tree automata. However, no witness generation algorithm has yet been formally described. We contribute a first, direct algorithm for JSON Schema witness generation. We study its effectiveness and efficiency, in experiments over several schema collections, including thousands of real-world schemas. Our focus is on the completeness of the language (where we only exclude the “ uniqueItems ” operator), on the ability of the algorithm to run in reasonable time on a large set of real-world examples, despite the exponential complexity of the problem, and on proving its correctness and completeness.
Infinite probabilistic databases (PDBs) are a framework of probability distributions over infinitely many different database instances. We systematically study the representability problem for such PDBs by means of tuple-independence and first-order views. Although first-order views over tuple-independent PDBs are not a complete representation system for infinite PDBs, they form a fairly robust class: Adding first-order constraints does not give them additional expressive power, and they cover many relevant special cases such as block-independent disjoint PDBs, and PDBs of bounded instance size. We identify criteria for representability (or non-representability) in this class and explore their limits. In addition, we study the expressive power of fragments of first-order logic over tuple-independent PDBs. In general, for infinite PDBs, the landscape of relative expressive power for various classes of views over PDBs with independence assumptions is much more complex than in the finite setting.
Finding frequent subgraph patterns in a big graph is an important problem with many applications such as classifying chemical compounds and building indexes to speed up graph queries. Since this problem is NP-hard, some recent parallel and distributed systems have been developed to accelerate the mining. However, they often have a huge memory cost, very long running time, suboptimal load balancing, poor scale-out capability, and possibly inaccurate results. In this article, we propose an efficient system called T-FSM for parallel mining of frequent subgraph patterns in a big graph. T-FSM supports a new anti-monotonic frequentness measure called Fraction-Score, which is more accurate than the widely used MNI measure. The execution engine of T-FSM supports both intra-machine parallelism and inter-machine parallelism. For intra-machine parallelism, T-FSM adopts a novel task-based execution model to ensure high multithreading concurrency, bounded memory consumption, and effective load balancing. For inter-machine parallelism, T-FSM ensures good scale-out performance with a lightweight pattern rebalancing approach that reduces workload skewness of pattern evaluations among machines. To avoid recomputing the contexts for migrated patterns, we design a novel context cache table to support concurrent and asynchronous requesting and caching of remote context data, which can timely evict and garbage collect used pattern contexts that are no longer needed to keep memory consumption bounded. Extensive experiments show that T-FSM is orders of magnitude faster than existing state-of-the-art parallel systems (more than 10 & times;, 51 & times;, 131 & times;, 55 & times; speedup over ScaleMine, DistGraph, Pangolin
Hierarchical directed acyclic graph (DAG) is an essential model for representing terminologies and their hierarchical relationships, such as Disease Ontology and ImageNet categories. Due to the vast number of terminologies and complex structures in a large DAG, it becomes challenging for humans to effectively analyze and explore the hierarchical relationships they encode. Therefore, summarizing hierarchical DAGs is essential for enhancing the interpretability and visualization of the underlying hierarchy. Beyond visual data exploration, hierarchical DAG summarization also supports a range of applications, such as biomedical ontology analytics, snippet generation for information search, and summarized recommendation. In this article, we address a new problem of finding k representative vertices to summarize a hierarchical DAG. To capture diverse summarization and identify important vertices, we design a summary score function that reflects vertices diversity coverage and structure correlation. The studied problem is theoretically proven to be NP-hard. To tackle it efficiently, we propose a greedy algorithm with an approximation guarantee that iteratively adds vertices with significant summary contributions to the answers. To further enhance the answer quality, we introduce a subtree extraction-based method that is proven to achieve higher-quality answers. Additionally, we develop a scalable algorithm, k-PCGS, which employs candidate pruning and DAG compression for large-scale hierarchical DAGs. For the query-dependent problem, we propose an index-based method and several optimization techniques to improve efficiency. Extensive experiments on large real-world datasets demonstrate the effectiveness and efficiency of our proposed algorithms.
The chase procedure is a fundamental algorithmic tool in databases that allows us to reason with constraints, such as existential rules, with a plethora of applications. It takes a database and a set of constraints as input and iteratively completes the database as dictated by the constraints. A key challenge, though, is the fact that the chase may not terminate, which leads to the problem of checking whether it terminates given a database and a set of constraints. In this work, we focus on the semi-oblivious version of the chase, which is well-suited for practical implementations, and linear existential rules, a central class of constraints with several applications. In this setting, there is a mature body of theoretical work that provides syntactic characterizations of when the chase terminates, algorithms for checking chase termination, precise complexity results, and worst-case optimal bounds on the size of the result of the chase (whenever it is finite). Our main objective is to experimentally evaluate the existing chase termination algorithms with the aim of understanding which input parameters affect their performance, clarifying whether they can be used in practice, and revealing their performance limitations. Concerning guarded existential rules, a natural generalization of linear existential rules, one can reuse the machinery for linear existential rules by first applying the so-called linearization technique, that is, the technique of converting guarded existential rules into linear existential rules without affecting the termination of the chase. A secondary objective of this work is to understand how realistic is the use of the linearization technique in the context of the semi-oblivious chase termination problem.
The introduction of Bitcoin fueled the development of blockchain-based resilient data management systems that are resilient against failures, enable federated data management, and can support data provenance. The key factor determining the performance of such resilient data management systems is the consensus protocol used by the system to replicate client transactions among all participants. Unfortunately, existing high-throughput consensus protocols are costly and impose significant latency on transaction processing, which rules out their usage in responsive high-performance data management systems. In this work, we improve on this situation by introducing the Proof-of-Execution consensus protocol ( PoE ), a consensus protocol designed for high-performance low-latency resilient data management. PoE introduces speculative execution , which minimizes latency by starting execution before consensus is reached, and PoE introduces proof-of-executions to guarantee successful execution to clients. Furthermore, PoE introduces a single-round check-commit protocol to reduce the overall communication costs of consensus. Hence, we believe that PoE is a promising step towards flexible general-purpose low-latency resilient data management systems.
The performance and scalability of modern data-intensive systems processing large datasets are limited by unnecessary data movement. Even though near-data processing (NDP) can provably reduce data transfers and increase performance, at present, NDP is utilized primarily in read-only settings. Near-data execution of data-intensive modification operations is currently infeasible due to the lack of transactional consistency and the absence of practicable low-latency synchronization mechanisms between the host database engine and the NDP-engine on smart storage. In this paper, we introduce Update NDP as an approach to offloading modifications to computational storage with transactional guarantees in an NDP database system called neoDBMS. To ensure consistency, we introduce a low-latency shared lock table between the host and computational storage, based on novel cache-coherent interconnects. We also introduce a novel locking protocol that seamlessly integrates the shared lock table within the lock manager of the host NDP-engine. To handle failure recovery, while preserving high and robust performance, we introduce novel extended locking and logging mechanisms that allow the host and computational storage to perform useful work during log-movement. Our evaluation indicates that in-storage modifications in neoDBMS in mixed workload settings are ≥ 6.52 × faster than host-only executions and exhibit robust performance due to lower data movement and better resource utilization.
The prevalence of computer graphics technology boosts the development of point clouds, which offer advantages over Triangular Irregular Networks, i.e., TINs, in proximity queries. All existing on-the-fly shortest path query algorithms and oracles on a TIN are expensive, and no algorithms can answer shortest path queries on a point cloud directly. Thus, we propose two types of efficient shortest path oracles on a point cloud. They answer the shortest path query between (1) a pair of Points-Of-Interests (POIs), and (2) any point and a POI, respectively. We propose four adaptations of them to answer the query between any point and a POI (or any point if no POIs are given). We also propose two efficient proximity query algorithms using these oracles. Our two oracles and their proximity query algorithms outperform the best-known adapted oracle by 12 to 42,000 times in terms of the oracle construction time, oracle size and proximity query time, respectively1.
The development of data analytics services has fueled many optimizations in data scans, and indexes are one of the most important techniques to improve scan efficiency. Meanwhile, block-based data organization has become standard practice in these services, providing an opportunity for more fine-grained index selection at the block level. However, today’s systems ignore data distribution differences among blocks and usually tune indexes over the entire database table, leading to unnecessary storage costs and potential degradation in query performance. To bridge this gap, we propose BISLearner, a fast, block-aware index selecting approach based on reinforcement learning. One major challenge lies in differentiating the data distribution among data blocks. To solve this problem, BISLearner maintains simplified histograms that represent the data distribution of each block. When a query is issued, BISLearner leverages the query predicate and histogram-based block summaries to generate a specific workload representation for each block. However, such block-aware workload representation leads to an excessive number of input features, resulting in a slow or even incorrect convergence of neural networks. Inspired by the human-learning process, where more attention is devoted to the important parts of data, we design an attention-based neural model to efficiently handle the high volume of input features caused by table partitioning and select the best-suited index combinations at the block level. Additionally, to handle the expensive search space caused by attribute combinations and data partitioning, we employ heuristic-based invalid action masking at the block level to accelerate the training process. Our evaluation using PostgreSQL and Greenplum database systems demonstrates BISLearner is able to reduce job completion time by up to 28.45% compared to its best counterparts.
Generation-based testing techniques have proven effective in detecting logic bugs in DBMS, often stemming from the improper implementation of query optimizers. However, existing generation-based debugging tools predominantly rely on random testing, which tends to overlook critical error-prone areas such as multi-table joining and grouped aggregation. In this article, we propose TQS, a novel testing framework targeted at detecting logic bugs arising from SPJG (Select-Project-Join-Group By) query optimizations. Given a target DBMS, TQS achieves the goal with two key components: Data-guided Schema and Query Generation (DSG) and Knowledge-guided Query Space Exploration (KQE). DSG addresses the key challenge of multi-table query debugging: how to generate ground-truth (query, result) pairs for verification. DSG utilizes data derived from dimensionally aggregated data cubes, which store data of grouped metric columns. It maps data from data cubes to a wide table, applies database normalization techniques to the wide table to generate a testing schema and maintains a bitmap index for result tracking. To improve debug efficiency, DSG also artificially inserts some noises into the generated data. To avoid repetitive query space search, KQE guides the generation of error-prone cubes, and forms the problem as isomorphic graph set discovery and combines the graph embedding and weighted random walk for query generation. We evaluated TQS on four popular DBMSs: MySQL, MariaDB, TiDB, and PolarDB. Experimental results show that TQS is effective in finding logic bugs of SPJG query optimization in database management systems. It successfully detected 226 bugs within 24 hours, including 63 bugs in MySQL, 52 in MariaDB, 68 in TiDB, and 43 in PolarDB, respectively.
Recent work has demonstrated the catastrophic effects of poor cardinality estimates on query processing time. In particular, underestimating query cardinality can result in overly optimistic query plans which take orders of magnitude longer to complete than one generated with the true cardinality. Cardinality bounding avoids this pitfall by computing an upper bound on the query's output size using statistics about the database such as table sizes and degrees, i.e., value frequencies. In this article, we extend this line of work by proving a novel bound called the Degree Sequence Bound, which takes into account the full degree sequences and the max tuple multiplicity. This work focuses on the important class of Berge-Acyclic queries for which the Degree Sequence Bound is tight and provably improves on prior work. We further describe how to practically compute this bound using a functional approximation of the true degree sequences and prove that even this functional form improves upon previous bounds. Lastly, we outline the challenges of implementing this in a real system and some techniques for overcoming these challenges.
An increased and growing interest in large-scale data processing has triggered a demand for specialized algorithms that thrive in massively parallel shared-nothing systems. To answer the question of how to efficiently compute join queries in this setting, a rich line of research has emerged specifically for the Massively Parallel Communication (MPC) model. In the MPC model, algorithms are executed in rounds, with each round consisting of a synchronized communication phase and a separate local computation phase. The main cost measure is the load of the algorithm, defined as the maximum number of messages received by any server in any round. We study worst-case optimal algorithms for the join query evaluation problem in the constant-round MPC model. In the single-round variant of MPC, the worst-case optimal load for this problem is well understood and algorithms exist that guarantee this load for any join query. In the constant-round variant of MPC, queries can often be computed with a lower load compared to the single-round variant, but the worst-case optimal load is only known for specific classes of join queries, including graph-like and acyclic join queries, and the associated algorithms use very different techniques. In this paper, we propose a new constant-round MPC algorithm for computing join queries. Our algorithm is correct for every join query and its load matches (up to a polylog factor) the worst-case optimal load for at least all join queries that are acyclic or graph-like.
Despite the wide use of k -Nearest Neighbors as classification models, their explainability properties remain poorly understood from a theoretical perspective. While nearest neighbors classifiers offer interpretability from a ''data perspective'', in which the classification of an input vector x is explained by identifying the vectors v 1 , ..., v k in the training set that determine the classification of x, we argue that such explanations can be impractical in high-dimensional applications, where each vector has hundreds or thousands of features and it is not clear what their relative importance is. Hence, we focus on understanding nearest neighbor classifications through a ''feature perspective'', in which the goal is to identify how the values of the features in x affect its classification. Concretely, we study abductive explanations such as ''minimum sufficient reasons'', which correspond to sets of features in x that are enough to guarantee its classification, and counterfactual explanations based on the minimum distance feature changes one would have to perform in x to change its classification. We present a detailed landscape of positive and negative complexity results for counterfactual and abductive explanations, distinguishing between discrete and continuous feature spaces, and considering the impact of the choice of distance function involved. Finally, we show that despite some negative complexity results, Integer Quadratic Programming and SAT solving allow for computing explanations in practice.
In this paper, we study the complexity of evaluating Conjunctive Queries with negation (\cqneg). First, we present an algorithm with linear preprocessing time and constant delay enumeration for a class of CQs with negation called free-connex signed-acyclic queries. We show that no other queries admit such an algorithm subject to lower-bound conjectures. Second, we extend our algorithm to Conjunctive Queries with negation and aggregation over a general semiring, which we call Functional Aggregate Queries with negation (\faqneg). Such an algorithm achieves constant delay enumeration for the same class of queries but with a slightly increased preprocessing time, which includes an inverse Ackermann function. We show that this surprising appearance of the Ackermmann function is probably unavoidable for general semirings but can be removed when the semiring has a specific structure. Finally, we show an application of our results to computing the difference of CQs.
Sampling over joins is a fundamental task in large-scale data analytics. Instead of computing the full join results, which could be massive, a uniform sample of the join results would suffice for many purposes, such as answering analytical queries or training machine learning models. In this paper, we study the problem of how to maintain a random sample over joins while the tuples are streaming in. Without the join, this problem can be solved by some simple and classical reservoir sampling algorithms. However, the join operator makes the problem significantly harder, as the join size can be polynomially larger than the input. We present a new algorithm for this problem that achieves a near-linear complexity. The key technical components are a generalized reservoir sampling algorithm that supports a predicate, and a dynamic index for sampling over joins. We also conduct extensive experiments on both graph and relational data over various join queries, and the experimental results demonstrate significant performance improvement over the state of the art.
