Edge servers usually store collected data in cloud servers and use deduplication techniques to remove redundancy. However, edge servers can also perform deduplication during data collection. This requires transferring fingerprints from the cloud servers to the edge servers for assistance. Since the large volume of fingerprint data on the cloud server, for example, 1 PB of data corresponds to 8 TB of fingerprints, transferring all fingerprints to the edge servers is impractical. Therefore, we propose a fingerprint offloading strategy. Only a small amount of fingerprints and data chunks needed for edge deduplication are offloaded from the cloud server to the edge server, enabling cloud-edge collaborative deduplication. The general process is as follows: First, the edge server collect a large amount of data from various devices, divides the data into chunks, and calculates fingerprints to identify unique data chunks and fingerprints. Then, the edge server upload the fingerprints to the cloud server. The cloud server check the fingerprints and offload the data chunks corresponding to existing fingerprints back to the edge server. Upon receiving these data chunks, the edge server performs thorough deduplication. Finally, the deduplicated data chunks are uploaded to the cloud server, ensuring that only unique data is transmitted to the cloud server. Experiments used chunks ranging from 1 KB to 16 KB, with an average size of 4 KB, and employed three real backup datasets. The results showed that the edge server computation time was reduced by 48.1%, metadata storage was reduced by 98.1%, and the upload volume from the edge server to the cloud server decreased by 87.1%. The size of fingerprints and data chunks offloaded to the edge server ranged from 9.4 MB to 1624.2 MB.
With advantages of scalability and low cost, cloud object storage has become the preferred storage solution for massive data. However, object storage has its own defects, such as poor performance of small I/Os, which restricts object storage from being widely used in some scenarios like Big Data. In this paper, we propose GooseFS, a distributed high-performance cache service built on top of object storage, which accelerates the access to object storage under the storage-compute separation architecture. GooseFS makes great enhancements on small I/Os performance as well as metadata performance for data stored in object storage. GooseFS introduces three key designs: (1) Compute-Side Cache, which reduces the data access latency under random small I/O with multi-granularity cache management and short circuit read mechauism. (2) Storage-Side Cache, which improves the throughput with a high-performance SSD cache pool. (3) Metadata Acceleration, which significantly improves the performance of metadata operations through special metadata organization and lock-free strong consistency cache. Experiment show that compared with native object storage, GooseFS increases the throughput of small random I/O by 5x -9 x and improves metadata operation performance by at least 3.9 x, meeting the performance requirements of various workloads.
Solid State Drive cache, implemented as on-board shared DRAM memory, can significantly enhance I/O performance by caching frequently accessed data. Although SSD caching strategies for single I/O data flows have been extensively explored, studies on cache partitioning to optimize parallel I/O in an SSD are scarce. In this paper, we present a novel dynamic cache partitioning approach designed to improve overall performance of multi-parallel I/O data flows by minimizing performance degradation of cache pollution and resource contention. By dynamically adjusting cache partition sizes for each data flow by considering cache sensitivity on performance, our strategy seeks to determine the optimal cache partition sizes to maximize overall I/O throughput. We implemented the strategy in the SSD simulator MQSim and evaluated its performance using various synthetic and real-world workloads. Our experimental results indicate that our dynamic cache partitioning strategy achieves an overall throughput increase of up to 33.22% compared to shared cache methods and outperforms static cache partitioning strategies by up to 21.19%.
As a complementary technique to data deduplication, delta compression calculates and stores differences between highly similar but non-duplicate chunks to reduce storage costs in storage systems. A key aspect of efficient post-deduplication delta compression is resemblance detection, which identifies similar reference chunks to achieve significant data reduction. However, our evaluation of existing methods indicates that the throughput of resemblance detection is the system bottleneck due to the computationally intensive nature of feature extraction, which is carried out sequentially. Additionally, these methods suffer from low similarity between detected chunks and miss some potential similar chunks, reducing the redundancy elimination effect. This paper introduces ERD, a technique aimed at optimizing resemblance detection, featuring two key advancements. First, it employs AVX-512 instructions to generate rolling hashes in parallel based on a novel overlapping sub-chunk partition strategy, and optimizes the linear transformation process, thereby reducing computational overhead while eliminating computational errors introduced by parallelization. Second, it introduces a locality-aware feature matching technique to further detect potentially similar chunks and enhance the detection capabilities. Experimental results show that ERD accelerates feature extraction for resemblance detection by 2.1x similar to 2.18x (the state-of-the-art method Finesse) and 5.66x similar to 6.21x (N-trans) while increasing the delta compression ratio by up to 32.5% (18.2% on average) (Finesse).
Node failures in decentralized distributed storage systems are common. To ensure data availability, these systems employ data redundancy mechanisms, typically relying on repli-cas. This paper proposes a decentralized data redundancy scheme based on erasure coding, with Reed-Solomon and IPFS as exam-ples. Compared with replication, the proposed scheme reduces storage space and enhances fault tolerance. Files are sharded and encoded across multiple nodes, avoiding the high redundancy of replicas. Users can retrieve any K shards from N nodes to reconstruct the original files. This erasure coding optimization combines efficient data exchange among decentralized nodes with erasure coding technology, significantly reducing storage space compared with the replica mechanism. The implementation involves truncating and sharding the blocks within the Merkle DAG generated by files, enabling flexible adjustments to the code rate of erasure codes and the allocation of storage nodes based on user needs and available resources. This method achieves a balance between storage efficiency and data availability.
