RocksDB is an embedded, high-performance, persistent key-value storage engine developed at Facebook. Much of our current focus in developing and configuring RocksDB is to give priority to resource efficiency instead of giving priority to the more standard performance metrics, such as response time latency and throughput, as long as the latter remain acceptable. In particular, we optimize space efficiency while ensuring read and write latencies meet service-level requirements for the intended workloads. This choice is motivated by the fact that storage space is most often the primary bottleneck when using Flash SSDs under typical production workloads at Facebook. RocksDB uses log-structured merge trees to obtain significant space efficiency and better write throughput while achieving acceptable read performance. This paper describes methods we used to reduce storage usage in RocksDB. We discuss how we are able to trade off storage efficiency and CPU overhead, as well as read and write amplification. Based on experimental evaluations of MySQL with RocksDB as the embedded storage engine (using TPC-C and LinkBench benchmarks) and based on measurements taken from production databases, we show that RocksDB uses less than half the storage that InnoDB uses, yet performs well and in many cases even better than the B-tree-based InnoDB storage engine. To the best of our knowledge, this is the first time a Log-structured merge tree-based storage engine has shown competitive performance when running OLTP workloads at large scale.

Oracle Database Replay has been recently introduced in Oracle 11g as a novel tool to test relational database systems [9]. It involves recording the workload running on the database server in a production system, and subsequently replaying it on the database server in a test system. A key feature of workload replay that enables realistic reproduction of a real workload is synchronization. It is a mechanism that enforces specific ordering on the replayed requests that comprise the workload. It affects the level of request concurrency and the consistency of the replay results when compared to the captured workload. In this paper, we define the class of consistent replay synchronization schemes and study, for the first time, the spectrum they cover and the tradeoffs they present. We place the only scheme proposed so far [9], the one implemented in Oracle 11g Release 1, within the aforementioned spectrum and show that it is coarse-grained and more restrictive than necessary, often enforcing dependencies between calls that are independent. By enforcing needless waits, it decreases the level of possible concurrency and degrades performance. To overcome these drawbacks, we identify the best scheme within the spectrum; it is finer-grained than its counterparts and strikes the right balance across different tradeoffs: it enforces a partial ordering on the replayed calls that minimizes the number of required waits and maximizes the level of concurrency, without compromising consistency of the replay results. We have implemented the new scheme in Oracle 11g Release 2. Our experiments indicate that it produces better quality replays than the pre-existing one for major classes of workload.

This demonstration presents Oracle Database Replay, a novel approach to testing changes to the relational database management system component of an information system (software upgrades, hardware changes etc). Database Replay makes it possible to subject a test system to a real production workload, which helps identify all potential problems before implementing the planned changes on the production system. Any interesting workload period of a production database system can be captured with minimal overhead. The captured workload is used to drive a test system while maintaining the concurrency and load characteristics of the real production workload. The demonstration showcases how important maintaining the concurrency and load characteristics of the real workload is. The current testing solutions do not allow for synchronization based on data dependencies. Without proper synchronization the demonstration workload does not perform the work required and does not exercise the test system appropriately, leading to poor coverage and inadequate load. Thus many issues remain undetected. Database replay with its data based synchronization makes testing realistic and leads to the discovery of potential problems.

Oracle Database Replay provides a new way to test changes to a database system by reproducing the real user workload in a test environment. It helps to identify potential problems after software or hardware upgrades, patches, or changes to database parameters, schema or data. Any interesting workload period of a production database system can be captured with minimal overhead. The captured workload can be used to drive a test system while maintaining the concurrency and load characteristics of the real production workload. The replay does not depend on any other software including the application itself. It reliably reproduces the captured workload to support early diagnosis and troubleshooting. In this paper we discuss the analysis of replay results and introduce a new compare-period report, which assists with a detailed performance comparison of the capture and its replays with respect to changes. We demonstrate its usefulness in a case study involving an upgrade for a Siebel financial application, where Database Replay identifies performance problems after the upgrade and helps correcting them.

This paper presents the Oracle solution for database testing delivered in the Oracle 11g Database Real Application Testing option. The first feature, the SQL Performance Analyzer, performs unit testing of the SQL of a production workload. The second feature, Database Replay, makes it possible to subject a test system to a real production system workload. Both features base their effectiveness on the fact that the best possible workload for testing is the actual production workload, which is unparalleled to our knowledge. By running the real application workload on a test system, these features can predict the impact of changes to a production system accurately and without risk. Combined with Oracle's performance manageability offerings, such as the Automatic Database Diagnostic Monitor (ADDM) and SQL Tuning Advisor, these features provide a compelling end-to-end approach for introducing change into production environments without risk of performance regression.

Computing aggregates over distributed data sets constitutes an interesting class of distributed queries. Recent advances in peer-to-peer discovery of data sources and query processing techniques have made such queries feasible and potentially more frequent. The concurrent execution of multiple and often identical distributed aggregate queries can place a high burden on the data sources. This paper identifies the scalability bottlenecks that can arise in large peer-to-peer networks from the execution of large numbers of aggregate computations and proposes a solution. In our approach peers are assigned the role of aggregate computation maintainers, which leads to a substantial decrease in requests to the data sources and also avoids duplicate computation by the sites that submit identical aggregate queries. Moreover, a framework is presented that facilitates the collaboration of peers in maintaining aggregate query results. Experimental evaluation of our design demonstrates that it achieves very good performance and scales to thousands of peers.

This thesis develops techniques for scalable data-centric distributed systems. The first part presents techniques for the ad-hoc formation of networks of data sources that allow scalable query processing. To quantify the performance of existing flooding-style query processing relative to the proposed techniques, a P2P system was built and evaluated. A node can join an existing P2P network by contacting any node already in the network. When a flooding-style query processing strategy is followed, nodes do not exchange data-specific information during network joining. Otherwise, a node provides some summary information about its data that is forwarded to all other peers. This information is stored in special structures, the peer indices, and is used to determine the relevant data sources for a query. Experiments show that peer indices on nodes is necessary for good performance. The second part presents the Catalog Service for mapping queries to data sources, which is based on Distributed Hash Tables (DHTs). Peers provide catalog information when they join the network. In the case of XML repositories each peer provides for each element (and attribute) a list of the paths that lead to it and an optional value summary. Thus, given an XPath query and the catalog information, one can determine the data sources that need to be accessed in order to process the query. Additionally, request load balancing methods are presented that make the Catalog Service scalable. The third and final part of this thesis explores distributed aggregate computations in data-centric P2P networks. They are important queries in large distributed systems because they allow the summarization of large distributed amounts of data. When an aggregate query is popular among the peers, data sources receive a large number of identical requests, which limits scalability. To address this problem the design of the Aggregation Layer is presented that assigns peers the maintenance of aggregates computations. Each maintainer acts as a computation hub by receiving updates from the data sources, whenever data changes, and by answering query requests. Simulation and system experiments prove the scalability and feasibility of the Aggregation Layer.