Evaluating Lightweight Integer Compression Algorithms in Column-Oriented In-Memory DBMS.

Lightweight data compression algorithms are often used to decrease memory consumption of in-memory databases. In recent years, various integer compression techniques have been proposed that focus on sequential encoding and decoding and exploit modern CPUs’ vectorization capabilities. Interestingly, another dominant access pattern in databases systems has seen little attention: random access decoding. In this paper, we compare end-to-end database performance for various integer compression codecs on three recent CPU architectures. Our evaluation suggests that random access performance is often more relevant than vectorization capabilities for sequential accesses. Before integrating selected encodings in the database core, we benchmarked seven libraries in an exhaustive standalone comparison. We integrated the most promising techniques into the relational in-memory database system Hyrise and evaluated their performance for TPC-H, TPC-DS, and the Join Order Benchmark on three different CPU architectures. Our results emphasize the importance of random access decoding. Compared to state-of-the-art dictionary encoding in TPC-H, alternatives allow reducing memory consumption of integer columns by up to 53 % while improving runtime performance by 5 % on an Intel CPU and over 16 % on an Apple M1. 1 COMPRESSION IN IN-MEMORY DATABASE SYSTEMS With increasing volumes of data being collected, the need for fast and efficient processing of workloads increases continuously. When customers’ workloads have substantial performance requirements that the Database Management System (DBMS) needs to serve, storing data on disk is often too slow. Technological advances in the main memory industry have made large main memory capacities affordable, enabling the adoption of in-memory databases [17]. For these DBMS, disk access is no This work is licensed under the Creative Commons BY-NC-ND 4.0 International License and appears in ADMS 2021, 12th International Workshop on Accelerating Analytics and Data Management Systems, August 16, 2021, Copenhagen, Denmark. longer the bottleneck. Instead, main memory access and processing efficiency are the new optimization goals [29]. Therefore, customers often choose to use in-memory databases to serve their performance-critical workloads. However, for in-memory databases, a compact representation of data is even more crucial. In a recent survey by the market researcher IDG [19], 80 % of the participating organizations stated that they have at least one part of their infrastructure running in the cloud and spend around one-third of their IT budget on cloud computing. Here, the bill depends on the used resources, which is often measured in RAM size1. To adapt in-memory databases to run efficiently on cloud infrastructures, therefore, means that they should use as few main memory as possible. Data compression can help to achieve this. However, several practical considerations limit the usage of compression. In addition to the strong performance requirements, Service Level Agreements (SLAs) in enterprise contexts often bound the maximum allowed query response time [30], which makes the usage of heavyweight compression schemes (e.g., LZ4 [9], LZ77 [38]) difficult. As an alternative, lightweight encoding schemes can still 75