Semi-structured data, encompassing formats such as logs, JSON files, and XML documents, is proliferating due to its flexible, partially organized nature. This growth outpaces advancements in storage technology, creating challenges for data-intensive systems where storage costs and random access performance are critical concerns. Existing compression techniques, primarily designed for general-purpose or block-level data, fail to leverage the inherent structural patterns in semi-structured data, resulting in suboptimal compression ratios or inefficiencies in lookup operations. To address these limitations, we propose Pattern-Based Compression (PBC), which specifically targets patterns in semi-structured data to achieve Pareto-optimality in most cases. Unlike traditional data block-based methods, PBC compresses data on a per-record basis, facilitating rapid random access. Empirical evaluation on real-world and benchmark datasets demonstrates that PBC achieves up to 50% better compression ratios than state-of-the-art methods while maintaining superior random access performance and competitive throughput. We also integrate PBC into a production database system and achieve improvements in both compression ratio and throughput.
原文链接
分享
引用
