Erasure coding is widely adopted to maintain data reliability, yet it introduces a significant update penalty. We analyze real-world traces and observe several challenges that are not addressed by existing studies, which thereby restricts the performance gains. We propose FastUpdate , an efficient multi-stripe updates framework that assists existing update schemes for fast updates. FastUpdate comprises three key designs: (i) it perceives the update locality and carefully merges multiple update requests accessing the same stripe to reduce the incurred network traffic; (ii) it abstracts the existing update schemes into collector selection and tree construction, greedily generates the update solution for each stripe to balance the transmission load across nodes; (iii) it dynamically schedules appropriate stripes to update in heterogeneous and dynamic networks to fully utilize the bandwidth resources. Comprehensive evaluations verify the effectiveness of FastUpdate on Alibaba ECS. It can increase the update throughput by 16.15%-88.71% for various update schemes.
Emerging Zoned Namespace (ZNS) SSDs transfer garbage collection (GC) to the host, thereby providing more stable performance and extending device life. Although the ZNS SSD has these hardware advantages, each zone in the ZNS SSD must be written sequentially. File systems always need to implement additional mechanisms (IO Scheduling or modifications to the driver/interface) to meet the sequential write constraint of the ZNS SSD, while fully utilizing the high bandwidth of the ZNS SSD under the constraint. These additional mechanisms usually bring extra overhead or limitations and can also cause the scalability problem for the file system. Therefore, in the past, it was naturally assumed that the sequential write constraint of the ZNS SSD would complicate the design of the file system and have a negative impact on file system scalability. However, in this paper, we argue that the previous insight on the impact of the ZNS SSD on file systems is not comprehensive enough. We regard the ZNS interface as a double-edged sword for file system scalability and found that the sequential write constraint of the ZNS device also has a positive impact on file system scalability. Specifically, order-preserving write is essential for crash consistency in file systems but is the scalability bottleneck of file systems that are compatible with SSDs that support Zoned Namespaces. The analysis in this paper indicates that the sequential write constraint of the ZNS device provides an excellent opportunity to simplify and optimize the order-preserving write mechanism in file systems. Based on the new insight, we propose a scalable ZNS-compatible file system called ScaleZNFS. Contrary to previous work that only focuses on mitigating the limitations imposed by the ZNS interface on the file system, ScaleZNFS not only mitigates the negative impact of the ZNS interface on file system scalability through its multi-head logging design and the per-log IO scheduler, but also employs the ZNS-driven order-preserving write mechanism, which can leverage the sequential write constraint of the ZNS interface, to eliminate the scalability bottleneck associated with the order-preserving write. ScaleZNFS outperforms F2FS and XFS by up to 5.05 × and 2.94 ×, respectively, and it can maintain stable scalability with increasing threads.
Consumer flash storage uses block interfaces, leading to performance and endurance issues like large mapping table overhead and write amplification. While zoned abstraction solves these problems in enterprise devices, it introduces new challenges for consumer devices with limited volatile memory and suboptimal file system design. These challenges highlight the need for a better understanding and new designs for zoned storage in consumer devices. We propose ConZone+, a practical and accurate emulation framework designed for consumer-grade zoned flash storage. Unlike existing enterprise-focused emulators, ConZone+ models unique consumer constraints (e.g., limited mapping caches and write buffers, hybrid flash media) and hybrid mapping. Furthermore, to enable system-level emulation, ConZone+ supports block interface for metadata area in file system and provides several useful scripts to assist users. Finally, ConZone+ introduces several enhancements on the firmware logic, including a configurable per-chip command queue, variable zone sizes, and compatibility with non-power-of-two block sizes. Users can explore the internal architecture and management strategies of consumer-grade zoned flash storage and integrate their optimizations into system software with ConZone+. To evaluate the accuracy of ConZone+, we configure it referencing a mobile phone equipped with zoned flash storage and compare its performance to that. The evaluation results show that the average performance deviations are limited to 12.8% for writes and 10% for reads. In addition, we conduct several case studies using ConZone+ to explore the firmware design and the adaptability of the log-structured features of F2FS and the transaction processing of SQLite.
Storage engine is a crucial component in relational databases (RDBs). With the emergence of Internet services and applications, a recent technical trend is to deploy a Log-structured Merge Tree (LSM-tree) based storage engine. Although such an approach can achieve high performance and efficient storage space usage, it also brings a critical double-logging problem—In LSM-tree based RDBs, both the upper RDB layer and the lower storage engine layer implement redundant logging facilities, which perform synchronous and costly I/Os for data persistence. Unfortunately, such “double protection” does not provide extra benefits but only incurs heavy and unnecessary performance overhead. In this article, we propose a novel solution, called Passive Data Persistence Scheme (PASV), to address the double-logging problem in LSM-tree based RDBs. By completely removing Write-ahead Log (WAL) in the storage engine layer, we develop a set of mechanisms, including a passive memory buffer flushing policy, an epoch-based data persistence scheme, and an optimized partial data recovery process, to achieve reliable and low-cost data persistence during normal runs and also fast and efficient recovery upon system failures. We also comprehensively analyze the recovery inefficiencies inherent in the undetermined persistence speeds in real-world workloads, and propose a hybrid persistence scheme that incorporates virtual-flush-based recovery optimizations to achieve the optimal recovery performance when performing epoch-based persistence. We implement a fully functional, open-sourced prototype of PASV based on Facebook’s MyRocks. Evaluation results show that our solution can effectively improve system performance by increasing throughput by up to 49.9% and reducing latency by up to 89.3%, and it also saves disk I/Os by up to 42.9% and reduces recovery time by up to 4.8%. 1
Approximate Nearest Neighbor Search (ANNS) plays a key role in database and AI infrastructure. It exhibits extremely high memory intensity with a ~1:1 compute-to-memory access ratio. Commodity Processing-in-Memory (PIM) hardware, such as the first real-world commercial product (UPMEM), is promising for overcoming the memory wall in ANNS. However, its reuse of the system DDR bus prevents the CPU and PIM cores from accessing memory simultaneously. This necessitates batch scheduling in existing systems, which, in turn, leads to severe underutilization in two scenarios: (1) inter-batch, where PIM remains idle while the CPU is copying data, and (2) intra-batch, caused by uneven load distribution of PIM cores in a batch. This article proposes an efficient PIM-capable ANNS system named PIMANN . We observe that each PIM core has an additional, undocumented, and little-known control interface (originally used for control commands like launching PIM kernels), which could be retrofitted for fine-grained arbitration of DDR bus access. Thus, PIMANN can break the traditional batching scheduling paradigm and adopt a fine-grained scheduling paradigm . With this key idea, PIMANN introduces (1) a persistent PIM kernel to eliminate inter-batch idling, (2) per-PU query dispatching to balance real-time load across cores, and (3) microarchitecture-aware per-PU optimization to overcome inherent PU-level hardware constraints. Experiments show that PIMANN can boost throughput by 2.5-11.1× compared to existing ANNS systems on CPU or GPU. The implementation of PIMANN is available at https://github.com/cds-ruc/PIM-ANNS.
The All-Distances Sketch (ADS) is a powerful and theoretically-sound sketching scheme that captures neighborhood information in graphs for approximate processing. It enables high-accuracy estimation of many useful applications with a guarantee of accuracy and can significantly accelerate the execution times by orders of magnitude. However, ADS requires a substantial amount of space, that is, multiple times larger than the graph data. More seriously, existing studies mainly focus on managing ADSs in memory, posing an increasing challenge for users who aim to leverage ADS for large-scale graph processing, particularly in light of the exponential growth of real-world graphs nowadays. To this end, this article introduces Oasis, an O ut-of-core A pproximate graph SYS tem that brings the ADS technique into practical use by leveraging storage effectively. Specifically, Oasis offers a holistic framework that facilitates both ADS construction and estimation. For ADS construction, it allows users to adjust the memory usage based on the machine’s available memory and enable an efficient construction process. For ADS estimation, Oasis provides a user-friendly interface to easily execute the estimators while mitigating the impact of slow storage I/O. Evaluation results show that Oasis provides a practical graph processing solution with exceptional execution time and low memory usage, at the cost of a slight decrease in accuracy.
To manage storage heterogeneity with diverse characteristics in bandwidth and latency, existing approaches such as caching or tiering , employ a vertically-stacked design by placing faster devices on top of slower devices. However, as the performance between device characteristics becomes blurred with emerging storage media, the philosophy of hierarchy inevitably falls short of fully leveraging the combined bandwidth of multiple devices. In this article, we propose a horizontally structured storage architecture that leverages the combined capabilities of heterogeneous devices. We introduce PolyStore , a meta layer atop storage medium-optimized file systems that spans userspace and the OS, allowing applications to access multiple storage devices concurrently with transparent, fine-grained data placement. PolyStore maximizes cumulative storage bandwidth and reduces hardware and software bottlenecks without compromising important properties such as sharing and security. Our evaluations show that PolyStore achieves 1.11x - 9.38x performance gains for micro-benchmarks and 1.52x - 2.02x for real-world applications across various device configurations.
Nowadays, shingled magnetic recording (SMR) disks have emerged as an alternative option to conventional hard disk drives (HDDs) owing to their high storage density and cost-effectiveness. However, when multiple users compete for access to an SMR disk simultaneously, certain users may experience significant slowdowns, leading to unfairness due to the inherent characteristics of SMR disks. Specifically, as the available space on an SMR disk dwindles, some users may be required to wait while the disk performs a time-consuming clean operation (i.e., cleaning slowdown) to reclaim sufficient space. To provide fair management mechanisms in SMR storage stacks, we propose FairCS (Fairness-aware Cleaning Slowdown) that is a host-software solution designed to promote fairness among concurrent users accessing an SMR disk. Our method aims at enhancing users’ awareness of fairness by mitigating the impact of cleaning slowdowns on an SMR disk. Experimental results demonstrate that FairCS not only narrows the gap of minimum and maximum cleaning slowdowns, but also reduces long-tail latency, thereby enhancing overall system fairness.
LSM-tree-based key-value (KV) stores mainly employ sorting-based operations (e.g., flush and compaction) to manage the KV pairs on disk. Through analysis and experiments with RocksDB, we identify that the sorting operations cause critical issues of operation coupling, including intertwined resource consumption within an operation, interdependencies and contention among operations. These coupling problems lead to dependency in resource usage and are particularly exacerbated on hybrid storage devices, causing significant resource fragmentation and increased write stalls. Existing approaches to mitigating write stalls rely on either fixed differentiated data management or superficial scheduling of data sorting operations, but they fail to fundamentally address the resource usage dependency caused by operation coupling. In this paper, we propose DecouKV , a key-value store designed to break resource usage dependency and improve overall resource utilization on hybrid storage devices through operation decoupling and adaptive scheduling. Specifically, DecouKV decouples data sorting operations into CPU-intensive index merge tasks and I/O-intensive data append and data flush tasks by separating indexes from data files. It manages indexes using a merge-friendly skip list-based structure and stores data in append-only files. We propose an elastic capacity adjustment for higher levels to enable deeper decoupling. In addition, DecouKV employs a parameterized, queue-based scheduling framework that integrates resource monitoring, read/write performance balancing, and adaptive task dispatching to fully exploit available system resources. We implement DecouKV and conduct experimental evaluations. Compared to RocksDB, as well as state-of-the-art systems such as MatrixKV, PrismDB, SplitDB and ADOC, DecouKV improves CPU utilization by 25.4%–32.3%, increases throughput by 2.3–4.9 ×, and reduces tail latency by 74.3%–91.4% under write-intensive workloads. DecouKV also achieves a modest throughput improvement of 1.2–2.3 × under read-intensive workloads.
Emerging storage technologies, such as persistent memory and ultra-low latency SSD, enable the deduplication file system (DedupFS) to use non-cryptographic hash for fast fingerprinting. However, we find that the accelerated computation exposes another major performance penalty: the seemingly innocuous in-storage deduplication metadata maintenance incurs up to 38% overhead in the I/O path. We find the root cause is that deduplication incurs additional crash consistency overhead due to the maintenance of dedup-specific fingerprint-to-physical mappings. However, our insight, called deduplication-assisted crash consistency , shows that the deduplication mapping does not necessarily incur overhead, but rather can be exploited to improve performance. Our insight is based on two key observations: (1) The deduplication mapping can be merged with the file system logical-to-physical mapping, forming a logical-fingerprint-physical (LFP) mapping. Thus, we can persist deduplication metadata alongside file system metadata in a single I/O; (2) Furthermore, this mapping, containing fingerprint of the data block, can be leveraged to verify data integrity. Thus, we can achieve orderless data writes by verifying the integrity during recovery. We propose DarthFS to realize the efficiency of deduplication-assisted crash consistency. Using a series of techniques to manage data, metadata, and orderless I/O, DarthFS successfully leverages deduplication to improve file system I/O performance while retaining compatibility, effectiveness, and memory efficiency. Experiments across a range of synthetic and real-world workloads show that DarthFS outperforms existing file systems even under the workloads without duplicates, consistently outperforms existing DedupFSes, and can minimize deduplication-induced crash consistency overhead.
As data volumes continue to escalate, conventional index structures struggle to keep pace with high-performance demands. Learned indexing is a data structure that leverages machine learning models to predict the locations of data, which significantly improves search performance compared to traditional indexing methods. However, existing learned index schemes often focus solely on optimizing read and write operations, neglecting scalability and robustness in their index design. This oversight leads to significant performance variations across workloads with different characteristics. In this paper, we present SRIndex , a scalable and robust learned index, to address the abovementioned problems. SRIndex consists of the following design primitives: (i) It implements a model layer , which is optimized with a select number of keys to reduce asynchronous retraining time and accelerate query processes; (ii) It incorporates a transition layer to achieve the scalability of the index; and (iii) It introduces a leaf layer to provide efficient storage and high access efficiency. Extensive experimental results demonstrate that SRIndex outperforms state-of-the-art learned indexes by improving 1.27 × of write throughput under dynamic workloads.
The system architecture of contemporary supercomputers is growing increasingly intricate with the ongoing evolution of system-wide network and storage technologies, making it challenging for application developers and system administrators to manage and utilize the escalating complexity of supercomputers effectively. Moreover, the limited experience of application developers and system administrators in conducting insightful analyses of diverse High-Performance Computing (HPC) workloads and the resulting array of resource utilization characteristics exacerbate the challenge. To address this issue, we undertake a comprehensive analysis of six years’ worth of 40 TB data (comprising I/O performance data and job running information) from Sunway TaihuLight, with 41,508 nodes, currently ranked as the world’s 13th-fastest supercomputer. Our study provides valuable insights into operational management strategies for HPC systems (i.e., job hanging caused by heavy-load benchmark testing, job starvation caused by aggressive scheduling policies) and I/O workload characteristics (i.e., getattr operations spiking caused by massive access to grid files, and a large number of files accessed by many applications in a short period), shedding light on both challenges and opportunities for improvements in the HPC environment. This article delineates our methodology, findings, and the significance of this study. Additionally, we discuss the potential of our research for future studies and practice within this domain.
NVMe SSDs are becoming increasingly larger and faster, making disk partitioning between multiple tenants a prevalent practice. However, co-located applications with diverse I/O patterns often trigger contention within SSD internals. In this work, we explore the potential of large-zone ZNS SSDs, which expose internal states and enable finer-grained controls, to support high-performance disk partitioning in cloud, data center, and local cluster/server environments. To address this, we first conduct a comprehensive inter-operation interference study, examining the interplay between small and large reads/writes on large-zone ZNS SSDs, alongside the impacts of I/O command selection, OPEN zone allocation, and request size manipulation strategies. Second, guided by our five key observations, we propose ZNSlice, which is the first system tailored to optimize the utilization and isolation of large-zone ZNS SSDs for multi-tenancy. It implements operation-level isolation by integrating a suite of optimizations including efficient OPEN zone allocation and management, differentiated read/write handling, zone write mode selection, I/O request size manipulation, and mitigation of background operations overhead. Evaluations using 9 workloads on 2 commodity large-zone SSDs demonstrate that ZNSlice delivers high performance disk partitions via the block interface, natively supporting conventional file systems and existing workloads without modifications. The results confirm that ZNSlice simultaneously achieves significant improvements in multiple metrics, up to 40.2% higher disk utilization, 98.4% better isolation, and 61.4%/71.8% reductions in average/tail latency compared to the best baseline when TPC-C is shared with other workloads.
Existing external graph processing systems struggle with low I/O efficiency, high computational overhead, and substantial development costs on modern NVMe SSDs, due to their reliance on complex loading and computing models that transform random I/Os into sequential access. While in-memory graph systems with general-purpose memory-storage caches (like OS page cache or TriCache) offer improved support for fine-grained I/Os and simplified programming, they often fail to leverage specific graph access patterns, resulting in suboptimal performance. This paper aims to enhance the I/O efficiency of large-scale graph processing on NVMe SSDs. We first introduce a novel chunk-based graph representation model, featuring classified and hierarchical vertex storage and chunk layout optimization, to improve I/O utilization. Additionally, we present a latency-optimized access mechanism featuring user-space asynchronous I/O execution and hotness-aware chunk caching management to accelerate I/O and boost cache efficiency. Our prototype, ChunkGraph, enables flexible graph algorithm implementation and efficient execution. Experiments demonstrate ChunkGraph significantly outperforms existing external graph systems and in-memory graph systems using general-purpose cache subsystems, delivering several-fold speedups on overall performance and superior I/O efficiency.
Learned indexes serve as essential index structures within storage systems, yet their efficiency is constrained by the memory bandwidth and latency limitations inherent in traditional computer architectures. Processing-in-memory (PIM) technology presents a promising approach by embedding processing units directly into memory components. In this paper, we introduce PIMLex, a well-designed learned index utilizing PIM to address the challenges posed by memory-bound operations. PIMLex tackles the capacity constraints found in current PIM hardware by adopting a decoupled two-level architecture, which harnesses both the robust data processing power of PIM and the expansive capacity of conventional DRAM. Furthermore, the design incorporates a PIM-optimized model structure to reduce computational demands that are inefficient for PIM. Coupled with a hotness-aware replication strategy that promotes load balancing across multiple PIM modules, PIMLex achieves superior performance under diverse workload scenarios. We have implemented PIMLex on UPMEM, a commercially available PIM platform, demonstrating a throughput improvement of 2.2 × compared to the DRAM-based ALEX.
Certain NAND flash-based storage devices are not equipped with dynamic random access memory (DRAM) for holding the whole mapping table, due to the constraints of chip size and the cost, such as secure digital cards. Such DRAM-less flash memory can load only a subset of frequently accessed mapping table entries into a limited-capacity, on-board static random access memory (SRAM) cache, to expedite address translation. To improve the use efficiency of the SRAM cache, this paper proposes to prefetch mapping table entries into the cache according to their locality . Then, it can minimize the number of translation page reads at the flash array caused by loading the required entries, thus improving I/O performance. Specifically, we use the indicator of runs test to reflect the locality of mapping table entries on the same translation page. When processing a missed mapping entry, it determines whether adjacent mapping entries accompanying the missed one should be loaded into the cache or not, on the basis of the runs of the target translation page. Consequently, subsequent requests requiring access to these mapping entries can be quickly responded to with the cached ones, instead of reading the target translation pages. In addition, we support cache management based on the runs test of the mapping entries to further improve the cache hit ratio. Experimental results show that our proposal can increase the hit ratio of mapping table entries by 39.4 % and reduce overall I/O latency by 23.4 % on average, in contrast to state-of-the-art schemes.
Storage savings and data confidentiality are two primary yet conflicting goals in outsourced backup management. While deduplication-aware encryption has been extensively studied to make deduplication viable for encrypted data, it is incompatible with fine-grained delta and local compression for further storage savings. We present ShieldReduce, a secure outsourced storage system that aims for fine-grained shielded data reduction by applying deduplication, delta compression, and local compression to data in a trusted execution environment based on Intel SGX, so as to achieve high storage savings with security guarantees. To mitigate the I/Os of accessing base chunks for delta compression in SGX, ShieldReduce adopts bi-directional delta compression via a novel hybrid inline and offline compression design to maintain the physical locality of base chunks. It also offers a storage-prioritized offline compression mode for further storage savings and ensures crash consistency during data reduction. Evaluation on various backup workloads shows that ShieldReduce achieves significant speedups over a shielded baseline without bi-directional delta compression, while maintaining comparable storage savings to fine-grained data reduction for plain data.
Cloud block storage (CBS) provides virtual disks with block-level accessibility. The petabyte-scale CBS systems maintain trillions of block-mapping key-value entries as metadata to track the storage location of each virtual block. Although SSD-based KV stores have been widely adopted in cloud systems for their high efficiency and durability, current SSD-based schemes face significant challenges in achieving deterministic access latency for latency-sensitive metadata services. Our experimental observations indicate that the substantial long-tail latency is primarily caused by (1) I/O blocking due to internal tasks of SSDs including modern Zone Namespace SSDs; and (2) additional disk I/Os when querying high-level indexes across memory and SSDs under memory-constrained environments. In this paper, we propose an SSD-based SIndex to store trillions of block-mapping entries for latency-critical cloud block storage, which performs comprehensive latency optimization across storage I/O scheduling and high-level indexing. To prevent long-tail I/Os while avoiding intrusive device modifications, SIndex introduces an inter-SSD I/O scheduling mechanism based on read/write separation and SSD state transitions, which mitigates latency fluctuations induced by garbage collection on conventional SSDs and zone operations on Zone Namespace SSDs. Additionally, SIndex employs opportunistic I/O speculation and a concurrent request balancing mechanism to reduce read disturbance and I/O contention. To query the storage location of targeted block-mapping entries with bounded latency, SIndex proposes a memory-efficient high-level index incorporates with a static data layout, preventing time-consuming disk lookups by keeping the index in memory. We evaluate the SIndex prototype using a variety of benchmarks and real-world traces on commodity SSDs. The results demonstrate that SIndex outperforms RocksDB and other approaches by up to 11.4 × in tail latency, keeping the 99.99 th -percentile latency below 400µs.
