Scalability testing is a critical step in preparing a database system for deployment in production environments. However, developers are rarely granted access to full-scale real-world datasets due to privacy and security constraints. Instead, they must often rely on small, and sometimes redacted, samples. To conduct meaningful scalability evaluations, developers must interpret the semantics of the available data and generate large-scale synthetic datasets that faithfully preserve their statistical and structural properties. This paper proposes a tabular data augmentation method based on Generative Adversarial Networks (GANs), which enables the expansion of input tabular datasets to much larger scales while maintaining their original statistical characteristics and inter-table relationships. We have applied the proposed method to Japanese medical insurance claims data and demonstrate that the augmented datasets closely preserve the statistical distributions of the original records. Furthermore, we have conducted a scalability test using a commercial database engine, assessing both capacity utilization and query performance. Our findings suggest that GAN-based tabular data augmentation provides a viable path toward realistic and scalable database benchmarking in data-restricted environments.
MLPerf is the best-known industry standard AI benchmark. With buy-in from key industry players, fully defined performance and accuracy metrics it stands alone as the de facto standard AI benchmark. At the same time, submissions to MLPerf on new platforms are very difficult. A significant investment is needed for a new organization to generate results competitive with established submissions. In this contribution, we show how AMD was able to publish competitive and even leadership performance in the space of 1 year covering both MLPerf Inference and MLPerf Training on several Instinct GPU platforms. Focusing on a few sub-benchmarks and extensive optimizations enabled the initial submissions. Overall, we find that MLPerf has a high barrier to entry and cross-company alignment is necessary for successful submissions.
Synthetic data generation is essential for database development, allowing for the simulation, benchmarking and testing of the customer workload in scenarios where real data is unavailable due to privacy, scale or accessibility constraints. In practice, developers often have only the database schema and a handful of statistics, such as table cardinalities, value distributions, number of distinct values, TOP-N values, and range-based histograms. Despite this common setting, existing tools fall short: benchmark-specific generators (e.g., TPC-H, TPC-C) are rigid and tied to fixed schemas; UI-based tools like Mockaroo ignore relational semantics; and model-based approaches such as SDV require access to real data to learn patterns. We present DataGenX (part of the TiDB toolchain), a lightweight and flexible synthetic data generator designed for this under-served but common scenario. DataGenX operates directly on annotated SQL DDL, where comments are used to specify column-level statistics and inter-table relationships (e.g., foreign key join cardinalities). Then it generates synthetic data that conforms to these annotations, producing realistic and consistent datasets without requiring access to real data. DataGenX also supports multi-threaded row generation, where each thread produces a disjoint set of rows and writes to a separate file, enabling scalable and efficient data generation on multi-core systems. To validate the fidelity of DataGenX, we reimplemented the TPC-C and TPC-H schemas using annotated DDL and generated data using DataGenX. We compared the resulting datasets to those produced by the official TPC data generators and DataGenX that achieved fidelity 99
Database systems with hybrid data management support, referred to as HTAP or HOAP architectures, have gained popularity for performing near-real-time data analytics. Such systems first emerged in the relational world, and the CH-benCHmark (CH) was proposed in 2011 to evaluate such relational systems. A decade later, with NoSQL database systems gaining traction for new application development, we proposed a first document-oriented variant of CH – called CH2 – that was aimed at evaluating hybrid data platforms in the document database space. Here, we introduce CH2++, a major revision of CH2 that is significantly more suitable for evaluating systems that support JSON analytics. Like CH and CH2, this new CH2++ benchmark borrows from and extends both TPC-C and TPC-H. Differences from CH2 include a much more document-oriented schema, the inclusion of additional fields to exercise analytical systems’ column-oriented features, and a corresponding JSON “do over” of the CH2 queries. The paper also includes preliminary performance isolation and scaling results from running CH2++ against Capella Operational and Capella Columnar, which are Couchbase’s DBaaS cloud service offerings. The results highlight the value of CH2++ for evaluating JSON-targeted HOAP and HTAP platforms.
HammerDB is a leading open-source tool for database benchmarking, widely adopted across enterprise, cloud, and academic environments to evaluate transactional and analytic database performance. Originally developed as a personal open source project for testing the Oracle database on Linux, HammerDB has grown through a mission of accessibility, cross-platform support, and real-world relevance to become a trusted solution across the industry. This paper presents the story of HammerDB’s development, its architectural choices, major milestones, and evolution into a mature benchmarking suite with support for the most popular commercial and open source database systems. A key recent advancement is the release of HammerDB v5.0 in spring 2025, introducing a simplified installation with native packages for Linux and Windows, enhancing both performance and usability, furthering HammerDB’s goal of providing modern, portable and easy to use benchmarking tools. With its integration into the TPC-OSS subcommittee, HammerDB continues to bridge the gap between open source innovation and formal benchmarking standards. We also highlight ongoing performance collaborations with the database industry and outline future directions, including expanded workloads, cloud-native capabilities, and opportunities for community-driven contributions. This paper invites discussion on how open tools like HammerDB can shape the next era of collaborative database performance benchmarking.
