Scalability testing is a critical step in preparing a database system for deployment in production environments. However, developers are rarely granted access to full-scale real-world datasets due to privacy and security constraints. Instead, they must often rely on small, and sometimes redacted, samples. To conduct meaningful scalability evaluations, developers must interpret the semantics of the available data and generate large-scale synthetic datasets that faithfully preserve their statistical and structural properties. This paper proposes a tabular data augmentation method based on Generative Adversarial Networks (GANs), which enables the expansion of input tabular datasets to much larger scales while maintaining their original statistical characteristics and inter-table relationships. We have applied the proposed method to Japanese medical insurance claims data and demonstrate that the augmented datasets closely preserve the statistical distributions of the original records. Furthermore, we have conducted a scalability test using a commercial database engine, assessing both capacity utilization and query performance. Our findings suggest that GAN-based tabular data augmentation provides a viable path toward realistic and scalable database benchmarking in data-restricted environments.
MLPerf is the best-known industry standard AI benchmark. With buy-in from key industry players, fully defined performance and accuracy metrics it stands alone as the de facto standard AI benchmark. At the same time, submissions to MLPerf on new platforms are very difficult. A significant investment is needed for a new organization to generate results competitive with established submissions. In this contribution, we show how AMD was able to publish competitive and even leadership performance in the space of 1 year covering both MLPerf Inference and MLPerf Training on several Instinct GPU platforms. Focusing on a few sub-benchmarks and extensive optimizations enabled the initial submissions. Overall, we find that MLPerf has a high barrier to entry and cross-company alignment is necessary for successful submissions.
Synthetic data generation is essential for database development, allowing for the simulation, benchmarking and testing of the customer workload in scenarios where real data is unavailable due to privacy, scale or accessibility constraints. In practice, developers often have only the database schema and a handful of statistics, such as table cardinalities, value distributions, number of distinct values, TOP-N values, and range-based histograms. Despite this common setting, existing tools fall short: benchmark-specific generators (e.g., TPC-H, TPC-C) are rigid and tied to fixed schemas; UI-based tools like Mockaroo ignore relational semantics; and model-based approaches such as SDV require access to real data to learn patterns. We present DataGenX (part of the TiDB toolchain), a lightweight and flexible synthetic data generator designed for this under-served but common scenario. DataGenX operates directly on annotated SQL DDL, where comments are used to specify column-level statistics and inter-table relationships (e.g., foreign key join cardinalities). Then it generates synthetic data that conforms to these annotations, producing realistic and consistent datasets without requiring access to real data. DataGenX also supports multi-threaded row generation, where each thread produces a disjoint set of rows and writes to a separate file, enabling scalable and efficient data generation on multi-core systems. To validate the fidelity of DataGenX, we reimplemented the TPC-C and TPC-H schemas using annotated DDL and generated data using DataGenX. We compared the resulting datasets to those produced by the official TPC data generators and DataGenX that achieved fidelity 99
Database systems with hybrid data management support, referred to as HTAP or HOAP architectures, have gained popularity for performing near-real-time data analytics. Such systems first emerged in the relational world, and the CH-benCHmark (CH) was proposed in 2011 to evaluate such relational systems. A decade later, with NoSQL database systems gaining traction for new application development, we proposed a first document-oriented variant of CH – called CH2 – that was aimed at evaluating hybrid data platforms in the document database space. Here, we introduce CH2++, a major revision of CH2 that is significantly more suitable for evaluating systems that support JSON analytics. Like CH and CH2, this new CH2++ benchmark borrows from and extends both TPC-C and TPC-H. Differences from CH2 include a much more document-oriented schema, the inclusion of additional fields to exercise analytical systems’ column-oriented features, and a corresponding JSON “do over” of the CH2 queries. The paper also includes preliminary performance isolation and scaling results from running CH2++ against Capella Operational and Capella Columnar, which are Couchbase’s DBaaS cloud service offerings. The results highlight the value of CH2++ for evaluating JSON-targeted HOAP and HTAP platforms.
HammerDB is a leading open-source tool for database benchmarking, widely adopted across enterprise, cloud, and academic environments to evaluate transactional and analytic database performance. Originally developed as a personal open source project for testing the Oracle database on Linux, HammerDB has grown through a mission of accessibility, cross-platform support, and real-world relevance to become a trusted solution across the industry. This paper presents the story of HammerDB’s development, its architectural choices, major milestones, and evolution into a mature benchmarking suite with support for the most popular commercial and open source database systems. A key recent advancement is the release of HammerDB v5.0 in spring 2025, introducing a simplified installation with native packages for Linux and Windows, enhancing both performance and usability, furthering HammerDB’s goal of providing modern, portable and easy to use benchmarking tools. With its integration into the TPC-OSS subcommittee, HammerDB continues to bridge the gap between open source innovation and formal benchmarking standards. We also highlight ongoing performance collaborations with the database industry and outline future directions, including expanded workloads, cloud-native capabilities, and opportunities for community-driven contributions. This paper invites discussion on how open tools like HammerDB can shape the next era of collaborative database performance benchmarking.
A stash is a storage medium such as Dynamic Random Access Memory (DRAM), Solid State Disk (SSD), Hard Disk Drive (HDD), or Non-Volatile Memory (NVM). This paper presents a disaggregated transactional key-value (KV) store, DiStash, that governs KVs cross pools of stash types. It enables an application to use a single transaction to read and write different copies of one or more key-value pair across the different pools of stashes. It simplifies the application logic by (a) preventing undesirable race conditions that may cause copies of data across different stash pools to become inconsistent and/or (b) failures that may result in loss of key-value pairs. A configuration of DiStash may use a pool of stashes as either ephemeral or durable storage. The application dictates whether the content of its participating stashes are inclusive (replicated) or exclusive (tiered). We implement a DiStash by extending FoundationDB. We quantify the tradeoffs with its design decisions using microbenchmarks and eBay’s production workload. We open source our implementation at https://github.com/ebay-USC/DiStash .
The Transaction Processing Performance Council (TPC) is initiating the development of a new benchmark standard to address the emerging demands of Generative AI (GenAI) workloads. Building on the success of TPCx-AI, which established a comprehensive framework for evaluating end-to-end machine learning pipelines, this new benchmark aims to capture the unique computational and data-centric characteristics of GenAI systems. Generative AI models, including large language models (LLMs), diffusion models, and multimodal architectures, introduce novel challenges in data processing, model evaluation, and system scalability. The new benchmark is being designed to reflect real-world, production-scale GenAI scenarios. The benchmark will represent relevant end to end scenarios and full-stack evaluation pipeline, including data ingestion, inference, post-processing, and quality assessment. The TPC consortium is actively engaging with industry stakeholders to define representative workloads, including text summarization, code generation, image synthesis, and multimodal reasoning. The benchmark may incorporate a range of performance metrics critical to GenAI, including throughput, latency, model quality and price performance. Emphasis will also be placed on reproducibility, transparency, and auditability, in line with TPC’s established benchmarking principles. This paper outlines the challenges, design goals, methodologies and early architectural considerations for a new GenAI benchmark.
Role-Based Access Control (RBAC) is a widely used method for controlling access to data in data management systems, offering a scalable approach to enforcing security policies. Despite its broad adoption, RBAC implementations vary significantly across different systems, leading to inconsistencies in performance. Currently, there is no standardized benchmark for evaluating RBAC performance. In this study, we propose a benchmark that focuses on two core components: the creation of role hierarchies with varying structures and the access to securable objects and role metadata through SQL queries. We evaluate three systems—PostgreSQL, MariaDB, and Snowflake—across on-premise and cloud deployments. Our findings reveal notable differences in RBAC performance, driven by I/O demand, caching behavior, and system architecture. These results highlight the diversity of RBAC implementations and the need for a systematic approach to evaluate RBAC at scale.
Key-value stores are the backbone of many modern SQL- and NoSQL-based data systems, serving a variety of real-world applications. Despite their widespread adoption, existing key-value benchmarks fall short across multiple dimensions when accurately replicating complex and dynamic real-world workloads. For instance, state-of-the-art key-value benchmarks, such as YCSB, KVBench, and db_bench, are unable to (i) emulate dynamic workloads where the workload composition and distribution changes arbitrarily over time; (ii) generate composite keys with different prefix distributions; and (iii) generate workloads with varied degrees of data sortedness. These limitations result in inaccurate performance evaluations and limit the ability to understand how a commercial key-value store performs under dynamically shifting workloads. In this paper, we introduce Tectonic, a highly configurable and resource-efficient Rust-based key-value workload generator designed to model the temporal, structural, and dynamic properties of real-world workloads. Tectonic offers (i) fine-grained control over data access patterns for inserts, updates, merges, point and range queries, and point and range deletes; (ii) configurable composite key generation/selection strategies; (iii) dynamic workload generation where the workload properties change over time; and (iv) generation of workloads with user-specified data sortedness. Tectonic does so (v) at a 2 × higher throughput than the state-of-the-art, (vi) while recording up to 84 https://github.com/SSD-Brandeis/tectonic .
The demand for LLM inference is soaring, driven by widespread adoption of generative AI. Dr. Lisa Su recently projected the data center AI accelerator market to exceed 500B by 2028—with over 60
This panel discussion will address the top-of-mind subject of Agentic AI Systems. With the rise of Generative AI and associated Large Language Models (LLMs) and Foundation Models (FMs), the focus of research and practice is shifting to building AI Agents for targeted tasks. Furthermore, a collection of AI Agents are being stitched together to design and deploy Agentic AI systems for enterprise-level workflows and use cases. While the excitement around building such systems that employ LLMs and FMs as building blocks grows, the need to assess the performance of multi-agent systems is growing in importance as well. Metrics such as latency, throughput, accuracy and power consumption are, of course, necessary, but also not sufficient. Additional consideration must be given to the type of workflow, the criticality and resiliency of the use-case, and the target infrastructure for deployment. The panelists will bring their perspectives about building distributed autonomous Agentic AI systems, leveraging their experience spanning industry as well as academia. Factors crucial to developing a comprehensive framework for useful benchmarks will be discussed, with the goal of highlighting best practices for as well as limitations in current state of the art in AI system benchmarking.
Knowledge distillation offers a transformative pathway to developing powerful, yet efficient, small language models (SLMs) suitable for resource-constrained environments. In this paper, we benchmark the performance and computational cost of distilled models against their vanilla and proprietary counterparts, providing a quantitative analysis of their efficiency. Our results demonstrate that distillation creates a superior performance-tocompute curve. We find that creating a distilled 8B model is over 2,000 times more compute-efficient than training its vanilla counterpart, while achieving reasoning capabilities on par with, or even exceeding, standard models ten times its size. These findings validate distillation not just as a compression technique, but as a primary strategy for building state-of-the-art, accessible AI
The performance of database management systems (DBMS) is traditionally evaluated using benchmarks that focus on workloads with (almost) fixed record lengths. However, some real-world workloads in key/value stores, document databases, and graph databases exhibit significant variability in value lengths, which can lead to performance anomalies, particularly when popular records grow disproportionately large. Existing benchmarks fail to account for this variability, leaving an important aspect of DBMS behavior underexplored. In this paper, we address this gap by extending the Yahoo! Cloud Serving Benchmark (YCSB) to include an "extend" operation, which appends data to record fields, simulating the growth of values over time. Using this modified benchmark, we have measured the performance of three popular DBMS backends: MongoDB, MariaDB with the InnoDB storage engine, and MariaDB with the MyRocks storage engine. Our experiments alternate between extending values and executing query workloads, revealing significant performance differences driven by storage engine design and their handling of variable-sized values. Our key contribution is the introduction of a novel benchmarking approach to evaluate the impact of growing value sizes and isolate the effect of querying data with a distribution of data sizes from any cost associated with accessing data after a history of updates. This highlights the need for more representative benchmarks that capture the dynamic nature of real-world workloads, providing valuable guidance for both practitioners and researchers.
The rapid advancements in transformer-based language models have opened new possibilities for applications such as question answering and synthetic dataset generation. However, generating high-quality synthetic datasets can be computationally expensive and require significant resources. Fine-tuning a language model on a synthetic dataset can lead to improved performance on context-dependent questions, but it is crucial to evaluate the quality of these datasets beforehand. This study explores the benchmarking of synthetic natural language datasets generated with open-source language models for question answering applications. We fine-tune the same base model on each dataset independently and assess its performance on common benchmarks for question answering with context. Additionally, we examine metrics of the synthetic datasets themselves to identify potential indicators of downstream benchmark performance. Our results show that measuring the semantic similarity between questions is an indicator of domain diversity in a synthetic question-and-answer dataset. We also demonstrate that short answer lengths may indicate low quality chain-of-thought answers. Overall, Llama 3 8B Instruct performs most consistently well, of the models considered, in synthetic dataset generation. Our findings provide valuable insights for practitioners seeking to develop effective synthetic datasets for language-based applications.
Driven by advancements in model capabilities and ease of access, machine learning (ML) and artificial intelligence (AI) are increasingly applied across industry and government sectors. Traditionally, ML training and serving either relies on big external service providers such as AWS or MS Azure, or requires data to be transferred from databases or data lakes to local or cloud environments. Apart from dependencies on external ML frameworks, these type transfers not only introduce significant overhead but also pose risks to data security and data integrity. Integrating these technologies directly within database systems promises significant advantages, particularly for production environments. However, the performance and capability of database systems for various ML scenarios remain unclear. To address these uncertainties, this paper proposes transferring the TPCx-AI benchmark toolkit into PostgreSQL using the MADlib extension. This enables the entire ML pipeline-from data loading and preprocessing to training, scoring, and serving-to be executed within the database system. We present the implementation details and compare its performance with the traditional Python-based approach from the toolkit. The implementation focuses on traditional ML algorithms and does not include Deep Learning techniques. Our evaluation, leveraging the synthetic data generator PDGF and use cases provided by TPCx-AI, offers a comprehensive analysis of the benefits and shortcomings of in-database ML training with PostgreSQL and MADlib. On an aggregated level, it shows a comparable performance between both system for most use cases. While the Python approach excels at model training, PostgreSQL with Apache MADlib demonstrates superior performance in data processing and inference tasks.
In OLAP (Online Analytical Processing) environments, star schema outperforms 3NF (Third Normal Form). It offers simpler querying and superior performance, along with better tool support, making it analyst-preferred. Currently, the SSB benchmark is the main star schema benchmark but it has limited workload compared to other OLAP benchmarks like TPC-H. This paper introduces StarBench, a novel star schema benchmark that combines the advantages of star schema data model with the comprehensive workload of TPC-H. StarBench utilizes a 4-table denormalized model derived from TPC-H, effectively addressing the data gaps present in SSB. Data loading in StarBench is performed via insert-select operations from TPC-H tables. The workload remains consistent with TPC-H, incorporating SQL modifications to align with the new data model. We evaluated StarBench on PostgreSQL to demonstrate the benchmark's feasibility.
The Internet of Things (IoT) is revolutionizing industries by generating an unprecedented volume of time series data, making robust and high-performance time series databases crucial for effective data management and analysis. Time series data is pivotal in sectors such as energy, finance, and manufacturing, each with unique challenges and requirements, which complicates the use of a universal benchmark for evaluating databases in different domain applications. This paper discusses benchmarking methodologies in evaluating time series databases, with a focus on IoT environments. Key characteristics like high cardinality for handling IoT data are highlighted, along with the integration of Artificial Intelligence (AI) technologies for analytics and the applicability of the TPCx-IoT benchmark. Future improvements to benchmarks are also considered to address the evolving demands of IoT applications.
Stream processing systems are a fundamental component of modern data processing, enabling timely and efficient handling of streaming data. To assess and compare the capabilities of stream processing systems, various benchmarks have been proposed over the past years. Examples span a wide range of use cases, ranging from benchmarks for enterprise computing to social network analyses and IoT networks. These benchmarks are designed with different focuses and exhibit different characteristics during execution. In this paper, we review existing stream processing benchmarks and analyze them across five dimensions: benchmark type, included workloads, data ingestion, supported systems under test (SUT), and tracked metrics. We compare their similarities and differences, providing a comprehensive overview of existing benchmarks. Finally, we discuss aspects that have been overlooked and highlight those that should be addressed when benchmarking future generations of streaming systems.
Hybrid Transaction/Analytical Processing (HTAP) databases are gaining popularity in the database community. The growing prominence of HTAP databases necessitates the development of suitable benchmarks tailored to evaluate these systems effectively. We believe existing HTAP benchmarks are not representative, as most of them merged the existing benchmarks for transactional and analytical workloads. Our research also found that the latest proposals did not take into account the online aspect of an HTAP database. Most of the existing proposals re-used previous benchmark data models that are not truly representative of HTAP workloads. This paper introduces Web3Bench, an HTAP benchmark that addresses the limitations mentioned above in the existing HTAP benchmarks. The benchmark we propose in this paper is based on a realistic workload in the Web3 domain, considered a predominant use case for an HTAP database. Our data model is a simplified version of the decentralized blockchain Ethereum dataset. In this benchmark, we leverage a sample data set from Ethereum to build a scale factor-based data generator. The workload in Web3Bench focuses on simple queries representing online processing with a high number of queries per second. In this paper, we also present the outcomes of our proof of concept conducted with this benchmark on the TiDB cluster, a distributed SQL database with HTAP features.
The proliferation of the Internet of Things (IoT) has resulted in an immense volume of time-series data across various applications, necessitating efficient time-series databases (TSDB) to handle high-throughput ingestion and low-latency queries. The representative one is Apache IoTDB, which is an open-source, distributed TSDB that can be applied on both edge and cloud. Despite various TSDBs, an authoritative benchmark is required to verify their performance fairly. As the first industry-standard benchmark for measuring the performance of gateway systems, TPCx-IoT is the appropriate choice. In this paper, we use the TPCx-IoT to measure TimechoDB which is based on IoTDB, and demonstrate its superior performance in handling extensive data ingestion and complex query operations. The results show that TimechoDB outperformed the other TSDBs submitted to the TPC committee and we analyze key factors of TimechoDB's advantages. Furthermore, we offer insights and recommendations for future improvements in the TPCx-IoT benchmark, aiming to drive advancements in the management of large-scale IoT data.
