Existing database knob tuning methods evaluate the actual performance of each configuration by fully executing the entire workload. However, our experimental analysis reveals that this exhaustive execution approach significantly limits tuning efficiency, particularly when dealing with underperforming configurations. To address this issue, we propose ESTune, which is designed to early-stop the execution of poorly performing configurations. ESTune approximates the actual performance of these configurations using high-confidence predicted values generated from partially executed workload data and configuration knob settings. This strategy significantly reduces the evaluation time for underperforming configurations while maintaining the overall tuning effectiveness. The high-confidence predicted values are produced by a Hybrid Bayesian Neural Network (HBNN), which models the performance distribution with respect to different knob configurations. To address the challenge of limited training data commonly encountered in database knob tuning, ESTune integrates a Model-Agnostic Meta-Learning (MAML), thereby enhancing the few-shot learning capability of the HBNN. Extensive evaluations on a wide range of workloads consistently demonstrate that ESTune improves the tuning efficiency of existing methods.

Machine learning applications require timely access to fresh data from primary–backup databases to ensure accurate inference. Existing log replay strategies treat all updates equally and adhere to log-order dependencies, or only prioritize frequently accessed tables, resulting in high latency for prediction queries that usually access a small subset of attributes. Allowing immediate query execution can reduce latency but risks substantial accuracy degradation, as prediction models exhibit varying sensitivity even to minor data staleness. In this paper, we propose AALR, an accuracy-aware log replay strategy that accelerates data visibility for prediction queries while preserving inference accuracy. AALR prioritizes replay at the attribute level, enabling fine-grained replay aligned with query access patterns to avoid unnecessary delay. It leverages a learning-based model to quantify the relationship between data freshness and prediction accuracy, supporting adaptive replay decisions under diverse workloads. Furthermore, AALR introduces an epoch-based two-step replay mechanism, combining column-level parallel classification with row-level latest transaction retention to improve parallelism and resource utilization. Extensive experiments on multiple real-world datasets demonstrate that AALR significantly reduces data visibility latency for prediction queries while maintaining high prediction accuracy, outperforming state-of-the-art log replay strategies.

While compute-storage disaggregation provides elasticity to cloud databases, the prevalent single-primary model restricts write scalability and fault tolerance. Maintaining data consistency becomes a key challenge when multiple primaries handle concurrent writes, especially with hot data cached locally. This paper presents MTC (Multi-Primary Transaction Commit), a scalable and non-intrusive commit mechanism for multiprimary databases. The core idea of MTC is to decouple global transaction ordering from log synchronization. By enabling primary nodes to asynchronously prefetch Global Transaction Sequence Numbers (GTSNs) and assign them to locally completed transactions, MTC implements a highly efficient “check-thenreplicate” workflow. Following a pipelined, row-level conflict detection, validated transactions commit immediately with their logs broadcast asynchronously. Conflicting transactions, however, are rolled back locally with minimal overhead, fundamentally avoiding the network cost of broadcasting logs for transactions destined to abort. Our experimental results demonstrate that MTC significantly improves performance and scalability due to lower network overhead and more efficient conflict detection, even under high contention deployments.

Log-structured merge trees (LSM-trees) serve as the backbone of modern storage engines, yet their compaction process frequently becomes the primary performance bottleneck in production environments, compromising quality of service (QoS) in latency-sensitive applications. While LSM-tree based database systems expose numerous tunable parameters, existing tuning approaches focus exclusively on optimizing average-case throughput and latency, leaving compaction-induced performance bottlenecks unaddressed. In this work, we formulate robustness to compaction-induced performance variability as a novel constrained optimization problem, aiming to reduce performance variance without violating the throughput and latency service level agreements (SLAs). We propose MerTune, a knowledge-driven and constraint-aware Bayesian optimization framework designed for LSM-tree compaction tuning. MerTune models both the optimization objective and SLA constraints via Gaussian processes, while leveraging large language models (LLMs) to extract and group parameter knowledge from the database's official documentation. By integrating workload characterization, LLM-guided parameter selection, and knowledge-driven Bayesian exploration, MerTune improves robustness against compaction-induced performance fluctuations. MerTune has been applied to OceanBase database, and extensive evaluations under representative workloads demonstrate that it reduces throughput fluctuation by over 66.4%, latency fluctuation by over 45.5%, and CPU utilization fluctuation by over 17.1%, achieving superior optimization effectiveness and faster convergence compared to state-of-the-art tuning methods.

Large language models (LLMs) advance natural language (NL) interaction with databases by converting queries into SQL. However, users often lack familiarity with database schemas, making it difficult to express precise query requirements. To address this, we propose SQL-QMARS, a multi-agent Text-to-SQL framework designed to interactively clarify user intent. The system evaluates query vagueness using a three-layer metadata structure (theme, table, and field). Based on this evaluation, it dynamically triggers two flows: recommending multi-granular suggestions for vague queries and resolving ambiguities for clear ones. Furthermore, the system supports fusing external data to expand the knowledge source for query suggestions. The demonstration indicates that SQL-QMARS effectively guides users from vague to precise queries, improving the practicality of NL-based database interaction.

Flaky failure triage is crucial for keeping distributed database continuous integration (CI) efficient and reliable. After a failure is observed, operators must quickly decide whether to auto-rerun the job as likely flaky or escalate it as likely persistent, often under CPU-only millisecond budgets. Existing approaches remain difficult to deploy in this setting because they may rely on post-failure artifacts, produce poorly calibrated scores under telemetry and workload shifts, or learn from labels generated by finite rerun policies. To address these challenges, we present SCOUT, a practical state-aware causal online uncertainty-calibrated triage framework for distributed database CI. SCOUT uses only strict-causal features, including pre-failure telemetry and strictly historical data, to make online decisions without lookahead. Specifically, SCOUT combines lightweight state-aware scoring with optional sparse metadata fusion, applies post-hoc calibration to support fixed-threshold decisions across temporal and cross-domain shifts, and introduces a posterior-soft correction to reduce label bias induced by finite rerun budgets. We evaluated SCOUT on a benchmark of 3,680 labeled failed runs, including 462 flaky positives, and 62 telemetry/context features. Further, we studied the feasibility of SCOUT on TiDB v7/v8 and a large GitHub Actions metadata-only trace. The experimental results demonstrated its effectiveness and usefulness. We deployed SCOUT in the production environment, achieving an end-to-end P95 latency of 1.17 ms on CPU.

Integrating LLMs for data processing enables semantic querying but causes GPU memory bottlenecks and redundant computations. We present SemInfer, an acceleration system for batch semantic processing that treats the KV Cache as a semantic index, offloading pre-computed caches to host storage to eliminate redundancy. To reduce the index size, we propose a pruning strategy based on last-layer aggregated attention to accurately retain critical semantic tokens. Furthermore, we employ a pipeline mechanism to enable the asynchronous overlapping of CPU-GPU transmission and inference computation. This demonstration showcases the complete workflow of SemInfer on the IMDB dataset, achieving up to a 16.3x inference speedup over direct LLM inference and a 90

The SQL-based exploratory data analysis has garnered significant attention within the data analysis community. The emergence of large language models (LLMs) has facilitated the paradigm shift from manual to automated data exploration. However, existing methods generally lack the ability for cross-domain analysis, and the exploration of LLMs capabilities remains insufficient. This paper presents TiInsight, an SQL-based automated cross-domain exploratory data analysis system. First, TiInsight offers a user-friendly GUI enabling users to explore data using natural language queries. Second, TiInsight offers a robust cross-domain exploratory data analysis pipeline: hierarchical data context (i.e., HDC) generation, question clarification and decomposition, text-to-SQL (i.e., TiSQL), and data visualization (i.e., TiChart). Third, we have implemented and deployed TiInsight in the production environment of PingCAP and demonstrated its capabilities using representative datasets. The demo video is available at https://youtu.be/JzYFyYd-emI.

Isolation Levels (IL) act as correct contracts between applications and database management systems (DBMSs). The complex code logic and concurrent interactions among transactions make it hard to expose violations of various ILs stated by DBMSs. With the recent proliferation of new DBMSs, especially the cloud ones, there is an urgent demand for a general way to detect bugs violating various ILs. The core challenges come from the requirements of: (a) lightweight (verifying without modifying the application logic in workloads and the source code of DBMSs), (b) generality (verifying various ILs), and (c) efficiency (performing efficient verification on a long running workload). To this end, we propose a powerful and practical bug-finding tool Leopard. For lightweight, we propose to infer transaction dependencies based on the time intervals of operations collected from the client-side, without modifying the source code of DBMSs. For generality, based on a thorough analysis of existing concurrency control protocols, we summarize and abstract four mechanisms which can implement ILs in all commercial DBMSs we have investigated. For efficiency, we design a two-level pipeline to organize and sort massive time intervals in a time and memory conservative way; we propose a mechanism-mirrored verification to simulate the concurrency control protocols implemented in DBMSs for high throughputs; From experimental results, Leopard outperforms existing methods Cobra and Elle. In practice, Leopard has a superpower to verify various ILs on any workload running on all commercial DBMSs. Moreover, it has discovered 49 bugs undetected by other existing methods.

Distributed databases are widely used in various fields, such as financial services and e-commerce. These businesses generally exhibit characteristics of large-scale and rapid growth. However, these business systems often suffer from deadlocks that prevent them from operating normally for extended periods. Traditional deadlock detection methods face challenges in scalability and efficiency, especially as the number of nodes increases. Therefore, deadlock detection has always been a research area in distributed databases. In this paper, we introduce an efficient deadlock detection algorithm called HAWK, leveraging a Hierarchical Approach based on WorKload modeling. Our algorithm addresses these issues by constructing a dynamic hierarchical detection tree that adapts to transaction patterns, significantly reducing time complexity and communication overhead. HAWK first models the workload and generates a predicted access graph (PAG), transforming the problem of partitioning detection task in the basic hierarchical detection into partition detection zone (DZ) in the PAG by a graph-cutting algorithm. Then, leveraging the properties of strongly connected components (SCCs) and deadlock cycles, the SCC-cut algorithm naturally partitions the system-wide deadlock detection into multiple non-intersecting detection zones, thereby enhancing detection efficiency. We used the greedy SCC-cut algorithm to perform a more fine-grained partitioning of the complex PAG. Finally, by periodically sampling and updating the hierarchical structure, the algorithm remains responsive to dynamic workload variations, ensuring efficient detection. Our approach outperforms both centralized and distributed methods, offering a more efficient and adaptive solution. Extensive experimental results demonstrate the effectiveness of the HAWK algorithm, showing significant reductions in the duration of the deadlock and improved system throughput.

High-reliability distributed coordination services have become an indispensable part of modern large-scale distributed systems. Popular coordination services (e.g., ZooKeeper) adopt a single-writer design to provide a centralized service for managing system metadata, including various configuration information and data catalogs, and to provide distributed synchronization functions. With the continuous increase in metadata size and the scale of distributed systems, these coordination services gradually become performance bottlenecks due to their limitations in capacity, read and write performance, and scalability. To bridge the gaps, we propose FDBKeeper, a novel solution that enables scalable coordination services on distributed ACID key-value database systems. Our motivation is that transactional key-value stores (i.e., FoundationDB) meet the demands of performance and scalability required by large-scale distributed systems over coordination service. To leverage these advantages, coordination services can be implemented as an upper layer on top of distributed ACID key-value databases. Our experimental results demonstrate that FDBKeeper significantly outperforms ZooKeeper across key metrics. Additionally, FDBKeeper reduces hardware resource costs on average by 33% in the production environment, resulting in substantial monetary cost savings. We have successfully replaced ZooKeeper with FDBKeeper in the production-grade ClickHouse cluster deployment.

For real-time analysis of up-to-date data, hybrid transaction/analytical processing (HTAP) systems have been extensively studied. In general, three techniques play a critical role in HTAP systems, which are resource isolation, consistency model, and data sharing. However, there still lacks a benchmark suite that could comprehensively cover the three techniques. The core challenges come from the requirements of: (a) consistent workload resource consumption (provide workloads with the same computational complexity); (b) query-oriented freshness evaluation (focus on the degree of version staleness in the range of queried data); (c) precise data sharing efficiency measurement (catch the synchronization status accurately). In this paper, we propose Vodka to address the above challenges. For resource isolation, we formalize the change of query cardinalities under dynamic modifications, and manipulate the cardinalities of various query operators to ensure consistent query complexity comparisons under any data size. For consistency model, we design a column value grained version management strategy based on which query-oriented freshness is calculated. For data sharing, we design a lightweight point query driven method to check the synchronization status accurately. We finally conduct extensive experiments on three representative systems to justify our designs and provide insights for future system developments.

Exploratory data analysis (EDA), coupled with SQL, is essential for data analysts involved in data exploration and analysis. However, data analysts often encounter two primary challenges: (1) the need to craft SQL queries skillfully and (2) the requirement to generate suitable visualization types that enhance the interpretation of query results. Due to its significance, substantial research efforts have been made to explore different approaches to address these challenges, including leveraging large language models (LLMs). However, existing methods fail to meet real-world data exploration requirements primarily due to (1) complex database schema, (2) unclear user intent, (3) limited cross-domain generalization capability, and (4) insufficient end-to-end text-to-visualization capability. This paper presents TiInsight, an automated SQL-based cross-domain exploratory data analysis system. First, we propose a hierarchical data context (i.e., HDC), which leverages LLMs to summarize the contexts related to the database schema, which is crucial for open-world EDA systems to generalize across data domains. Second, the EDA system is divided into four components (i.e., stages): HDC generation, question clarification and decomposition, text-to-SQL generation (i.e., TiSQL), and data visualization (i.e., TiChart). Finally, we implemented an end-to-end EDA system with a user-friendly GUI in the production environment at PingCAP. We have also open-sourced all APIs of TiInsight to facilitate research within the EDA community. Through extensive evaluations by a real-world user study, we demonstrate that TiInsight offers remarkable performance compared to human experts. Additionally, TiSQL achieves an execution accuracy of 86.3% on the Spider dataset when using GPT-4. It also attains an execution accuracy of 60.98% on the Bird test dataset.

Update-intensive workloads are prevalent in contemporary OLTP and AI/ML scenarios. An update operation typically involves deleting the old version of the target record and then inserting a new version. In this work, we demonstrate that an LSM-tree faces two issues when dealing with update-intensive workloads. Firstly, the deleted old versions are not promptly garbage collected until they merge with their new versions during compaction. This may lead to space waste and write amplification. Secondly, it is common for an update operation to modify only a small fraction of a data record, such as one of a hundred attributes. However, state-of-the-art LSM-trees fail to effectively utilize the incremental storage strategy, which involves storing only the updated fraction rather than the entire new version to enhance efficiency. In this paper, we propose two techniques, active and fast garbage collection, and adaptive incremental updating, to address these issues, respectively. Active and fast garbage collection probes the distribution of invalid data versions in an LSM-tree and performs garbage collection in a more promptly manner. Adaptive incremental updating applies different storage modes to the update operation to achieve balanced write and read amplification ratios as much as possible. Based on the techniques, we introduce SylphDB implemented based on the codebase of RocksDB and optimized for update-intensive workloads. Experimental results demonstrated that, compared to traditional LSM-tree based systems, SylphDB can improve the efficiency of garbage collection by 2× and reduce write amplification by 20%.