MS scheduler: New, scalable, and high-performance sparse AVX-2 parity encoding and decoding technique for erasure-coded cloud storage systems

In cloud storage scenarios, each chunk with multiple symbols of a coding strip resides in a different node and, hence, on a different disk. Existing STAIR and sector-disk (SD) scheduling techniques are not scalable because the performance of existing STAIR and SD encoding and decoding schedulers decreases sharply as the number of parity disks increases. This paper proposes a new, scalable, multisector (MS) scheduler that provides three-level fault tolerance for multiple sectors, disks, and nodes. In addition, the MS scheduler is extendable to multilevel failure recovery. MS schedulers are aware of the inner structure of each node, such as the number of disks and sectors for each node. Compared with SD and STAIR schedulers, an MS scheduler improves the average encoding and decoding performance by 198.34%, on average (ranging from 170.46% to 228.54%). However, STAIR and SD schedulers are bound to a specific number of parity disks and sectors. This is because the MS scheduler uses streaming SIMD extensions (SSE) for the dense matrix-based parity computation software implementation. In contrast, an MS scheduler can perform parity calculations for a large number of parity disks and sectors using advanced vector extensions (AVX)-2, OpenMP thread parallelism, and sparse parity computation software implementation. As a result, fewer XOR operations are performed to calculate parity chunks owing to the diagonal structure of the MS scheduler. The sparse AVX-2/-512 diagonal erasure coding implementation of the MS scheduler enables the proposed scheduler to be compatible with recovery from multilevel failures by skipping a large number of XOR and multiplication operations. This proves that the MS scheduler is suitable for use in a scalable cloud storage environment.