There is an increasing need to support high-volume, concurrent transaction processing on shared data in both high-performance and datacenter computing. A recent innovation in server architectures is the adoption of disaggregated memory organizations utilizing the Compute eXpress Link (CXL) interconnect protocol. While CXL memory architectures alleviate many concerns in datacenters, enforcing ACID semantics for transactions in CXL memory faces many challenges. We describe a novel solution for supporting ACID (Atomicity, Consistency, Isolation, Durability) transactions in a CXL-based disaggregated shared-memory architecture. We call this solution HTCXL for Hierarchical Transactional CXL. HTCXL is implemented in a software library that enforces transaction semantics within a host, along with a back-end controller to detect conflicts across hosts. HTCXL is a modular solution that allows various combinations of HTM or software-based transaction management to be mixed as needed. We perform experimental evaluation of HTCXL using micro-architectural processor simulation and several STAMP benchmarks. Our method shows a significant speedup over a software approach on CXL fabric.

With the recent explosive growth in worldwide data and data processing demands, the need to support a large volume of transactions on shared data is increasing in both high performance computing and datacenter processing. A recent innovation in server architectures is the use of disaggregated memory based on the Compute eXpress Link (CXL) interconnect protocol. This memory architecture is increasing in popularity as it allows for dynamic demand-sensitive resizing of aggregated memory, support for heterogeneous memory types, and sharing of data amongst supported processors and devices, including computational accelerators. However, while this new memory architecture alleviates many concerns in datacenter architectures, the data integrity when using memory based transactions over CXL faces many challenges.To solve for these challenges, we describe a novel solution for providing ACID (Atomicity, Consistency, Isolation, Durability) transactions in a CXL-based disaggregated memory architecture. We call this solution Transactional CXL or TCXL. TCXL requires no changes to the existing processor microarchitectures and is implemented in a software library with a back-end controller that can be embedded in a CXL controller or as a stand alone CXL Device or implemented on a host.The transactions support persistent memory durable transactions and in-memory volatile transactions which can be in a pooled memory expansion for a single processor or shared amongst multiple processors. TCXL also supports processor based Hardware Transactional Memory (HTM) based transactions both on processor and over CXL. We evaluate TCXL by extending a CXL simulator and executing micro-benchmarks. In addition to gaining the benefits of using CXL, we show TCXL outperforms other approaches.

With the recent explosive growth in worldwide data processing demands, the need to support a large volume of transactions on shared data is increasing in both edge and datacenter processing. A recent innovation in server architectures is the use of disaggregated memory based on the Compute eXpress Link (CXL) interconnect protocol. Such memory architectures are increasingly popular in datacenters, as they allow dynamic demand-sensitive resizing of aggregated memory and support heterogeneous memory types. However, while this alleviates many concerns in datacenter architectures, ensuring data integrity when using memory based transactions over CXL faces many challenges. We describe a novel hardware-based scheme for providing ACID (Atomicity, Consistency, Isolation, Durability) transactions in a CXL-based disaggregated memory architecture. The architecture and protocol presented in this paper requires only simple hardware extensions, and is applicable to both persistent memory durable transactions or in-memory volatile transactions.

Throughout the modern computing era, the complexity of high-performance computing systems has increased exponentially, both in terms of the number of nodes contained within a system and the number of transistors. This exponential complexity increase introduces probabilities for system failures that may affect a single node or element on a node. If these failures are not properly mitigated, a single failure may disrupt or halt the entire system.To address these potential failures, modern systems implement a form of checkpointing, whereby the system state is saved and restarted if a failure occurs. However, synchronous checkpointing an entire system pre-failure can be costly, requiring nodes to wait, often causing a significant decrease in system application performance.In this work, we introduce Continuous Checkpointing for HPC Systems, creating an asynchronous continuous checkpoint, and if a failure occurs, revert to a consistent global checkpoint. We utilize emerging Compute eXpress Link (CXL) Memory to lazily post computational state to nearby nodes and allow users to trade space for time efficiently. Our methods show a significant increase in system utilization over traditional checkpointing, and our lightweight method performs close to an unsafe system with no checkpoints.

Recent developments in non-volatile memory and networking technologies raise new challenges and opportunities for architecting storage systems. In this paper we propose Telepathy, a lightweight data access protocol for NVRAM+RDMA-based distributed storage systems. Telepathy is a fully distributed protocol whose I/O writes can be coordinated by any server node, and I/O reads can be served by any of the replicas. Telepathy guarantees strongly-consistent reads while providing high I/O concurrency. Hybrid RDMA operations are used to transmit data directly and efficiently to the NVRAM of target servers. The correctness of Telepathy is verified with a formal proof of consistency, and its performance is validated with YCSB benchmarks on the Chameleon cluster. Telepathy can achieve low I/O latencies and high throughput, with low CPU utilization.

This paper describes a new resource allocation policy for partitioning the cycles of a multiprocessor among concurrent multi-threaded jobs, together with a novel scheduler implementation in the Linux kernel. Each job consists of tasks whose execution times are drawn from some discrete distribution. Our policy shifts the established application-centric allocation to task-centric allocation by treating the server as a set of logical servers and dividing their joint resources fairly. We maximize task throughput while maintaining client fairness. We designed and developed the OPT scheduler for the Linux kernel, and obtain significant gains in resource utilization over CFS for various workloads. Our OPT scheduler adapts to dynamic workloads and can schedule multi-threaded clients efficiently to implement our resource allocation policy.

Advances in persistent memory and networking hardware are changing the architecture of storage systems and data management services in datacenters. Distributed, one-sided RDMA access to memory-resident data shows tremendous improvements in throughput, latency and server CPU utilization of storage servers. However, the silent nature of one-sided I/O simultaneously creates new challenging problems for providing QoS in such systems. In this paper, we propose Haechi, a work-conserving, token-based QoS mechanism to guarantee reservations and limits in storage systems that provide one-sided I/O services. Haechi decouples QoS enforcement into a QoS engine at the client and a QoS monitor at the data node. It leverages adaptive token dispatch, token conversion, and silent I/O reporting to guarantee the reservations of distributed clients while maintaining high server utilization. Empirical evaluations on the Chameleon cluster, with different reservation distributions and I/O access patterns, show that Haechi is successful in providing differentiated QoS with negligible overhead for token management.

Advances in storage, processing, and networking hardware are changing the structure of distributed applications. RDMA networks provide multiple communication mechanisms that enable novel hybrid protocols specialized to different data transfer requirements. In this paper, we present a distributed communication scheme that separates control and data communication channels directly at the RNIC rather than the application level. We develop a new communication artifact, a remote random access buffer, to efficiently implement this separation. Data messages are sent silently to the receiver, which is informed of the location of the data by a subsequent control message. Experiments on an RDMA-enabled cluster with micro benchmarks and two distributed applications validate the performance benefits of our approach.

Recent advances in hardware technologies raise new opportunities for architecting storage systems to exploit emerging NVRAM memory devices, fast remote-memory RDMA networking, and large numbers of processor cores. In this paper, we present Telepathy, a novel data access protocol for a distributed key-value storage system constructed around NVRAM and RDMA. Telepathy supports replicated data storage for fault tolerance and guarantees strong consistency while supporting high-volume concurrent read/write access from multiple client nodes. Our read protocol can perform largely silent consistent reads from any of the replica nodes holding an object, while our write protocol exploits remote atomic and non-volatile buffers to silently perform write contention resolution. Our experiments on an RDMA-enabled cluster using YCSB benchmarks show that Telepathy can achieve microsecond-range reads and writes with small tail latencies, GB-range data access bandwidth with small CPU utilization, and data consistency guarantees.

Providing performance QoS (Quality-of-Service) guarantees in a distributed data center environment poses unique challenges. In a typical setting, a client must be guaranteed a minimum amount of service (its contractual reservation) aggregated over all servers on which it has demand for service. A server may receive service demands from multiple clients, which may in aggregate exceed its service capacity. The system must decide how much service to provide to each client on each server that it has demand, in order to satisfy all clients' reservations. In case there is no feasible allocation of the service, the amount of reservation that is fulfilled must be maximized. In practice, the number of clients is several orders of magnitude larger than the number of servers. We describe a simple iterative algorithm called pTrans for reservation allocation based on a directed graph model; servers are vertices and edges are characterized by a vector of feasible service transfers between adjacent servers. We provide formal proofs that the algorithm converges to the global optimal and has a polynomial worst-case running time. Moreover, pTrans can be easily parallelized using multiple threads. Empirical evaluation results show pTrans has low runtime overheads.

Managing the trade-off between efficiency and fairness in a storage system is challenging due to high variability in workload behavior. Most workloads are made up of a mix of asymmetric operations (e.g. read/write, sequential/random, or striped/isolated I/Os) in different proportions, which places different resource demands on the storage device. The problem is to allocate device resources to the heterogeneous workloads fairly while maintaining high device throughput. In this paper, we present a new model for fair allocation of heterogeneous workloads with different ratios of asymmetric operations. We propose an adaptive scheme that chooses between two policies-the traditional Time-Balanced Allocation (TBA) and our proposed Bottleneck-Balanced Allocation (BBA)-based on workload characteristics. The fairness and throughput of these allocation policies are established through formal analysis. Our algorithms are tested with an adaptive, dynamic scheduler implemented in a simulation testbed, and the results validate the performance benefits of our approach.

We present Fair-EDF, a framework for latency guarantees in shared storage servers. It provides fairness control while supporting latency guarantees. Fair-EDF extends the pure earliest deadline first (EDF) scheduler by adding a controller to shape the workloads. Under overload it selects a minimal number of requests to drop and to choose the dropped requests in a fair manner. The evaluation results show Fair-EDF provides steady fairness control among a set of clients with different runtime behaviors.

The paper addresses the problem of providing performance QoS guarantees in a clustered storage system. Multiple related storage objects are grouped into logical containers called buckets, which are distributed over the servers based on the placement policies of the storage system. QoS is provided at the level of buckets. The service credited to a bucket is the aggregate of the IOs received by its objects at all the servers. The service depends on individual time-varying demands and congestion at the servers. We present a token-based, coarse-grained approach to providing IO reservations and limits to buckets. We propose pShift, a novel token allocation algorithm that works in conjunction with token-sensitive scheduling at each server to control the aggregate IOs received by each bucket on multiple servers. pShift determines the optimal token distribution based on the estimated bucket demands and server IOPS capacities. Compared to existing approaches, pShift has far smaller overhead, and can be accelerated using parallelization and approximation. Our experimental results show that pShift provides accurate QoS among the buckets with different access patterns, and handles runtime demand changes well.

Providing latency support is an important problem for clustered storage systems. In this paper, we present Fair-EDF, a framework for latency guarantees in shared storage servers. It provides fairness control while supporting latency guarantees. Fair-EDF extends the pure earliest deadline first (EDF) scheduler by adding a controller to shape the workloads. Under overload it selects a minimal number of requests to drop and to choose the dropped requests in a fair manner. The evaluation results show Fair-EDF provides steady fairness control among a set of clients with different runtime behaviors.

Emerging Persistent Memory technologies (also PM, Non-Volatile DIMMs, Storage Class Memory or SCM) hold tremendous promise for accelerating popular data-management applications like in-memory databases. However, programmers now need to deal with ensuring the atomicity of transactions on Persistent Memory resident data and maintaining consistency between the order in which processors perform stores and that in which the updated values become durable. The problem is specially challenging when high-performance isolation mechanisms like Hardware Transactional Memory (HTM) are used for concurrency control. This work shows how HTM transactions can be ordered correctly and atomically into PM by the use of a novel software protocol combined with a Persistent Memory Controller, without requiring changes to processor cache hardware or HTM protocols. In contrast, previous approaches require significant changes to existing processor microarchitectures. Our approach, evaluated using both micro-benchmarks and the STAMP suite compares well with standard (volatile) HTM transactions. It also yields significant gains in throughput and latency in comparison with persistent transactional locking.

With the emergence of new computing paradigms (e.g., cloud and edge computing, big data, Internet of Things (IoT), deep learning, etc.) and new storage hardware (e.g., non-volatile memory (NVM), shingled-magnetic recording (SMR) disks, and kinetic drives, etc.), a number of open challenges and research issues need to be addressed to ensure sustained storage systems efficacy and performance. The wide variety of applications demand that the fundamental design of storage systems should be revisited to support application-specific and application-defined semantics. Existing standards and abstractions need to be reevaluated; new sustainable data representations need to be designed to support emerging applications. To take advantage of hardware advancements, new storage software designs are also necessary in order to maximize overall system efficiency and performance. Therefore, there is a urgent need for a consolidated effort to identify and establish a vision for storage systems research and comprehensive techniques that provide practical solutions to the storage issues facing the information technology community. To address this need, the National Science Foundation's (NSF) Visioning Workshop on Data Storage Research 2025 brought together a number of storage researchers from academia, industry, national laboratories, and federal agencies to develop a collective vision for future storage research, as well as to prioritize near-term and long-term storage research and scientific investigations. In-depth discussions were carried out at the workshop along four major themes: (1) Storage for Cloud, Edge, and IoT Systems; (2) AI and Storage; (3) Rethinking Storage Systems Design; and (4) Evolution of Storage Systems with Emerging Hardware. The participants especially underscored the need for focused educational and training activities to instill storage system tools and technologies in the next generation of researchers and IT practitioners. Finally, the development of shared, scalable, and flexible community infrastructure to enable and sustain innovative storage research and verifiable evaluation was also discussed. This report presents the findings from these discussions.

We consider the problem of providing QoS guarantees in a clustered storage system whose data is distributed over multiple server nodes. Storage objects are encapsulated in a single logical bucket and QoS is provided at the level of buckets. The service that a single bucket receives is the aggregate of the service it receives at the nodes holding its constituent objects. The service depends on individual time-varying service demands and congestion at the physical servers. In this paper, we present bQueue, a coarse-grained scheduling algorithm that provides reservation and limit QoS for buckets in a distributed storage system, using tokens to control the amount of service received at individual storage servers. bQueue uses the max-flow algorithm to periodically determine the optimal token distribution based on the demands of the buckets at different servers and the QoS parameters of the buckets. Our experimental results show that bQueue provides accurate QoS among the buckets with different access patterns, and handles runtime demand changes in a reasonable way.

This paper addresses the problem of creating durable transactions in byte-addressable Non-Volatile Memory or Persistent Memory (PM) when using Hardware Transactional Memory (HTM)-based concurrency control. It shows how HTM transactions can be ordered correctly and atomically into PM by the use of a novel software protocol combined with a Persistent Memory Controller, without requiring changes to processor cache hardware or HTM protocols. In contrast, previous approaches require significant changes to existing processor microarchitectures. Our approach, evaluated using both micro-benchmarks and the STAMP suite compares well with standard (volatile) HTM transactions. It also yields significant gains in throughput and latency in comparison with persistent transactional locking.