Memory tiering provides a promising approach to enhance the flexibility of system memory configurations in modern compute servers. Intel® Flat Memory Mode (FLAT) is a hardware-managed memory tiering system for CXL memory that operates at 64 B cache-line granularity. In this paper, we evaluate the performance of FLAT using On-Line Analytical Processing (OLAP) workloads in SAP HANA, an in-memory database management system. Our results demonstrate that FLAT outperforms software-managed memory tiering and directly-attached CXL memory by leveraging cache-line granularity of data placement and exploiting data locality. For the first time, we evaluate FLAT with large-scale enterprise business warehouse workloads in SAP HANA, showing that it achieves performance comparable to a DRAM-only baseline, despite the high latency and limited bandwidth of CXL memory. Currently, FLAT accepts only a 1:1 ratio between local and CXL memory. When the local memory capacity exceeds that of CXL memory, Intel® Flat Memory Mode operates in a mixed mode (MIXED) containing flat memory and additional dedicated memory with remaining local memory in a separate NUMA node. Our evaluation of MIXED shows that performance degradation depends on access patterns and the degree of data locality exploited in flat memory. However, it requires software adjustments to utilize both dedicated memory and flat memory. To eliminate the need for such adjustments in MIXED, we propose a consolidated mode (CONSOLIDATED) that merges them into a single NUMA node. Our evaluation demonstrates that CONSOLIDATED results in negligible performance degradation, regardless of the ratio between dedicated memory and flat memory, and requires no software modifications at the application level.

Memory tiering is an effective strategy for enhancing the memory-to-core ratio in modern compute servers. Intel’s Flat Memory Mode (FLAT) represents a cutting-edge, hardware-managed memory tiering system for CXL memory, operating at cache-line (64B) granularity with yet unknown potential for database management systems. In this paper, we evaluate the performance of FLAT using OLAP workloads in SAP HANA, an in-memory database management system. Our findings demonstrate that flat memory outperforms directly attached CXL memory by leveraging data placement and exploiting locality. However, FLAT currently accepts only a 1:1 ratio between local and CXL memory. When the local memory capacity increases, FLAT operates in a mixed mode (MIXED) containing flat memory and additional dedicated memory with remaining local memory in a separate NUMA node. Our evaluation of MIXED shows that performance degradation depends on access patterns and the amount of exploited data locality when moving table data to flat memory or allocating heap memory in flat memory. To address the configuration complexity of two separate NUMA nodes in MIXED, we propose a new CONSOLIDATED mode that merges two NUMA nodes. Our evaluation shows that the consolidated mode results in negligible performance degradation, regardless of the ratio between dedicated memory and flat memory, and requires no software modifications at the application level.

CXL-based disaggregated memory systems offer options to expand the memory beyond the limits of a single server via cache-coherent memory expansion cards or memory pools. Especially, In-Memory Database Management Systems (IMDBMSs) can benefit from alleviating two critical constraints: (1) limited memory capacity in a server and (2) long restart time during failover to reload data to memory. However, the usage and effectiveness of CXL memory in enterprise-scale IMDBMSs has yet to be validated. In this work-for the first time-we investigate dynamic memory expansion employing commercial CXL memory devices for IMDBMSs. Our detailed performance analysis reveals that the performance impact of higher latency and lower memory bandwidth impact depends on the memory access patterns of data structures (cf. (1)). Additionally, we present the feasibility of CXL shared memory between servers to improve restart times during failover (cf. (2)). Our evaluation shows the effectiveness of CXL memory integrated into the SAP HANA Cloud IMDBMS. OLTP workloads have a negligible performance degradation while OLAP workloads have a wide range of performance degradation. CXL shared memory shows a 40% reduction of the restart time for TPC-H SF10 and 84% potential reduction for TPC-H SF100.

In the landscape of High-Performance Computing (HPC), the quest for efficient and scalable memory solutions remains paramount. The advent of Compute Express Link (CXL) introduces a promising avenue with its potential to function as a Persistent Memory (PMem) solution in the context of disaggregated HPC systems. This paper presents a comprehensive exploration of CXL memory's viability as a candidate for PMem, supported by physical experiments conducted on cutting-edge multi-NUMA nodes equipped with CXL-attached memory prototypes. Our study not only benchmarks the performance of CXL memory but also illustrates the seamless transition from traditional PMem programming models to CXL, reinforcing its practicality. To substantiate our claims, we establish a tangible CXL prototype using an FPGA card embodying CXL 1.1/2.0 compliant endpoint designs (Intel FPGA CXL IP). Performance evaluations, executed through the STREAM and STREAM-PMem benchmarks, showcase CXL memory's ability to mirror PMem characteristics in App-Direct and Memory Mode while achieving impressive bandwidth metrics with Intel 4th generation Xeon (Sapphire Rapids) processors. The results elucidate the feasibility of CXL memory as a persistent memory solution, outperforming previously established benchmarks. In contrast to published DCPMM results, our CXL-DDR4 memory module offers comparable bandwidth to local DDR4 memory configurations, albeit with a moderate decrease in performance. The modified STREAM-PMem application underscores the ease of transitioning programming models from PMem to CXL, thus underscoring the practicality of adopting CXL memory.

Efficient in-memory data processing relies on the availability of sufficient resources, be it CPU time or available main memory. Traditional approaches are coping with resource limitations by either adding more processors or RAM sticks to a single server (scale-up) or by adding multiple servers to a network cluster (scale-out). Further, the InfiniBand interconnect enables Remote Direct Memory Access (RDMA) and thus enhances the possibilities of resource sharing between distinct servers. Resource disaggregation means the (dynamic) sharing of available hardware, e. g., through the network. This paradigm is now further enhanced by the specification of Compute Express Link (CXL). In this paper, we systematically evaluate the implications of memory expansion as a form of resource disaggregation from the perspective of in-memory data processing through the local Ultrapath Interconnect (UPI), RDMA via InfiniBand, and PCIe attached memory via CXL. Our results show that CXL yields behavior that is comparable to UPI and outperforms the inherently asynchronous RDMA connection. Further, we found that handling UPI-attached memory as a type of disaggregated resource can yield additional performance benefits.

The separation and independent scalability of compute and memory is one of the crucial aspects for modern in-memory database systems (IMDBMSs) in the cloud. The new, cache-coherent memory interconnect Compute Express Link (CXL) promises elastic memory capacity through memory pooling. In this work, we adapt the well-known IMDBMS, SAP HANA, for memory pools by features of table data placement and operational heap memory allocation on far memory, and study the impact of the limited bandwidth and higher latency of CXL. Our results show negligible performance degradation for TPC-C. For the analytical workloads of TPC-H, a notable impact on query processing is observed due to the limited bandwidth and long latency of our early CXL implementation. However, our emulation shows it would be acceptably smaller with the improved CXL memory devices.

Non-volatile random-access memory (NVRAM) is about to hit the market and will require significant changes to the architecture of in-memory database systems. Since such hybrid DRAM-NVRAM database systems will keep the primary data solely persistent in the NVRAM, efficient replication mechanisms need to be considered to prevent data losses and to guarantee high availability in case of NVDIMM failures. In this paper, we argue for a software-based replication approach and present compute node-local mechanisms to provide the building blocks for an efficient NVRAM replication with a low latency and throughput penalty. Within our evaluation, we measured up to 10x less overhead for our optimized replication mechanisms compared to the basic replication mechanism of the Intel persistent memory development kit (PMDK).

Lightweight integer compression algorithms play an important role in in-memory database systems to tackle the growing gap between processor speed and main memory bandwidth. Thus, there is a large number of algorithms to choose from, while different algorithms are tailored to different data characteristics. As we show in this paper, with the availability of byte-addressable non-volatile random-access memory (NVRAM), a novel type of main memory with specific characteristics increases the overall complexity in this domain. In particular, we provide a detailed evaluation of state-of-the-art lightweight integer compression schemes and database operations on NVRAM and compare it with DRAM. Furthermore, we reason about possible deployments of middle- and heavyweight approaches for better adaptation to NVRAM characteristics. Finally, we investigate a combined approach where both volatile and non-volatile memories are used in a cooperative fashion that is likely to be the case for hybrid and NVRAM-centric database systems.

Persistent memory (PM) is fundamentally changing the way database index structures are built by enabling persistence, high performance, and (near) instant recovery all on the memory bus. Prior work has proposed many techniques to tailor index structure designs for PM, but they were mostly based on volatile DRAM with simulation due to the lack of real PM hardware. Until today is it unclear how these techniques will actually perform on real PM hardware. With the recent released Intel Optane DC Persistent Memory, for the first time, this paper provides a comprehensive evaluation of recent persistent index structures. We focus on B + -Tree-based range indexes and carefully choose four representative index structures for evaluation: wBTree, NV-Tree, BzTree and FPTree. These four tree structures cover a wide, representative range of techniques that are essential building blocks of PM-based index structures. For fair comparison, we used an unified programming model for all trees and developed PiBench , a benchmarking framework which targets PM-based indexes. Through empirical evaluation using representative workloads, we identify key, effective techniques, insights and caveats to guide the making of future PM-based index structures.

Storage Class Memory (SCM) has the potential to drastically change the database system landscape – much like high core count CPUs and large DRAM capacities spurred a shift to in-memory databases a decade ago. One of the possibilities provided by SCM is to significantly improve restart performance. SCM-enabled databases can evince a single-level main-memory architecture that stores, accesses, and modifies data directly in SCM, removing the traditional recovery bottleneck for main-memory databases: reloading data from durable media to main memory. Almost instantaneous recovery is possible, but at the cost of reduced throughput as latency sensitive data structures such as indexes need to be kept on SCM whose read and write latencies are projected to be noticeably slower than those of DRAM. We can regain this throughput by fully storing secondary data structures in DRAM and rebuilding them after restart, concurrently with processing incoming requests. While these data structures are being rebuilt, request throughput will be reduced not only because of the rebuilding overhead but, more significantly, because their (temporary) absence will result in sub-optimal access plans. Hence, rebuilding DRAM-based data structures becomes the new bottleneck for SCM-enabled databases recovery. In this paper, we address this bottleneck and describe Adaptive Recovery, a novel recovery technique that significantly reduces the impact on request throughput during the recovery period over naı̈ve approaches such as rebuilding data structures when first referenced.

Non-Volatile RAM (NVRAM) is a novel class of hardware technology which is an interesting blend of two storage paradigms: byte-addressable DRAM and block-addressable storage (e.g. HDD/SSD). Most of the existing enterprise relational data management systems such as SAP HANA have their internal architecture based on the inherent assumption that memory is volatile and base their persistence on explicit handling of block-oriented storage devices. In this paper, we present the early adoption of Non-Volatile Memory within the SAP HANA Database, from the architectural and technical angles. We discuss our architectural choices, dive deeper into a few challenges of the NVRAM integration and their solutions, and share our experimental results. As we present our solutions for the NVRAM integration, we also give, as a basis, a detailed description of the relevant HANA internals.

Storage Class Memory (SCM) is a novel class of memory technologies that promise to revolutionize database architectures. SCM is byte-addressable and exhibits latencies similar to those of DRAM, while being non-volatile. Hence, SCM could replace both main memory and storage, enabling a novel single-level database architecture without the traditional I/O bottleneck. Fail-safe persistent SCM allocation can be considered conditio sine qua non for enabling this novel architecture paradigm for database management systems. In this paper we present PAllocator, a fail-safe persistent SCM allocator whose design emphasizes high concurrency and capacity scalability. Contrary to previous works, PAllocator thoroughly addresses the important challenge of persistent memory fragmentation by implementing an efficient defragmentation algorithm. We show that PAllocator outperforms state-of-the-art persistent allocators by up to one order of magnitude, both in operation throughput and recovery time, and enables up to 2.39x higher operation throughput on a persistent B-Tree.

Storage Class Memory (SCM) is emerging as a viable solution to lift DRAM's scalability limits, both in capacity and energy consumption. Indeed, SCM combines the economic characteristics, non-volatility, and density of traditional storage media with the low latency and byteaddressability of DRAM. In this paper we survey research works on how SCM can be leveraged in databases and explore different solutions ranging from using SCM as disk replacement, to single-level storage architectures, where SCM is used as universal memory (i.e., as memory and storage at the same time), together with the challenges that stem from these opportunities. Finally, we synthesize our findings into recommendations on how to exploit the full potential of SCMin next-generation database architectures.

The advent of Storage Class Memory (SCM) is driving a rethink of storage systems towards a single-level architecture where memory and storage are merged. In this context, several works have investigated how to design persistent trees in SCM as a fundamental building block for these novel systems. However, these trees are significantly slower than DRAM-based counterparts since trees are latency-sensitive and SCM exhibits higher latencies than DRAM. In this paper we propose a novel hybrid SCM-DRAM persistent and concurrent B-Tree, named Fingerprinting Persistent Tree (FPTree) that achieves similar performance to DRAM-based counterparts. In this novel design, leaf nodes are persisted in SCM while inner nodes are placed in DRAM and rebuilt upon recovery. The FPTree uses Fingerprinting, a technique that limits the expected number of in-leaf probed keys to one. In addition, we propose a hybrid concurrency scheme for the FPTree that is partially based on Hardware Transactional Memory. We conduct a thorough performance evaluation and show that the FPTree outperforms state-of-the-art persistent trees with different SCM latencies by up to a factor of 8.2. Moreover, we show that the FPTree scales very well on a machine with 88 logical cores. Finally, we integrate the evaluated trees in memcached and a prototype database. We show that the FPTree incurs an almost negligible performance overhead over using fully transient data structures, while significantly outperforming other persistent trees.

The advent of Storage Class Memory (SCM) is driving a rethink of storage systems towards a single-level architecture where memory and storage are merged. In this context, several works have investigated how to design persistent trees in SCM as a fundamental building block for these novel systems. However, these trees are significantly slower than DRAM-based counterparts since trees are latency-sensitive and SCM exhibits higher latencies than DRAM. In this paper we propose a novel hybrid SCM-DRAM persistent and concurrent B-Tree, named Fingerprinting Persistent Tree (FPTree) that achieves similar performance to DRAM-based counterparts. In this novel design, leaf nodes are persisted in SCM while inner nodes are placed in DRAM and rebuilt upon recovery. The FPTree uses Fingerprinting, a technique that limits the expected number of in-leaf probed keys to one. In addition, we propose a hybrid concurrency scheme for the FPTree that is partially based on Hardware Transactional Memory. We conduct a thorough performance evaluation and show that the FPTree outperforms state-of-the-art persistent trees with different SCM latencies by up to a factor of 8.2. Moreover, we show that the FPTree scales very well on a machine with 88 logical cores. Finally, we integrate the evaluated trees in memcached and a prototype database. We show that the FPTree incurs an almost negligible performance overhead over using fully transient data structures, while significantly outperforming other persistent trees.

With the emergence of new hardware technologies, new opportunities arise and existing database architectures have to be rethought to fully exploit them. In particular, recovery mechanisms of current main-memory database systems are tuned to efficiently work on block-oriented, high-latency storage devices. These devices create a bottleneck during transaction processing. In this paper, we investigate the opportunities given by the upcoming Storage Class Memory (SCM) technology for database system recovery mechanisms. In contrast to traditional block-oriented devices, SCM is byte-addressable and offers a latency close to that of DRAM. We propose a novel main-memory database architecture that directly operates in SCM, eliminates the need for logging mechanisms, and provides a way to trade recovery time with the overall query performance. We implemented these concepts in our prototype SOFORT. Our evaluation shows that we are able to achieve instant recovery of the DBMS while removing the need for transaction rollbacks after failure.

In recent years, DRAM technology improvements have scaled at a much slower pace than processors. While server processor core counts grow from 33% to 50% on a yearly cadence, DDR 3/4 memory channel bandwidth has grown at a slower rate, and memory latency has remained relatively flat for some time. Combined with new computing paradigms such as big data analytics, which involves analyzing massive volumes of data in real time, there is a trend of increasing pressure on the memory subsystem. This makes it important for computer architects to understand the sensitivity of the performance of big data workloads to memory bandwidth and latency, and how these workloads compare to more conventional workloads. To address this, we present straightforward analytic equations to quantify the impact of memory bandwidth and latency on workload performance, leveraging measured data from performance counters on real systems. We demonstrate how the values of the components of these equations can be used to classify different workloads according to their inherent bandwidth requirement and latency sensitivity. Using this performance model, we show the relative sensitivities of big data, high-performance computing, and enterprise workload classes to changes in memory bandwidth and latency.

In recent years, DRAM technology improvements have scaled at a much slower pace than processors. While server processor core counts grow from 33% to 50% on a yearly cadence, DDR4 memory channel bandwidth has grown at a slower rate, and memory latency has remained relatively flat for some time. Meanwhile, new computing paradigms have emerged, which involve analyzing massive volumes of data in real time and place pressure on the memory subsystem. The combination of these trends makes it important for computer architects to understand the sensitivity of the workload performance to memory bandwidth and latency. In this paper, we outline and validate a methodology for quick and quantitative performance estimation using a real-world workload.