Erasure-tolerant coding is applied in data storage systems, and also for data transmission to tolerate the loss of data parts due to failures. In the paper we focus on a particular coding variant that we call compressed low-rate code. We report on properties of the code, draw requirements for an implementation, report on the compression results that we reached with our software-based system and provide a model for reliability assessment. A particular result is the necessity of a buffer management for data prior encoding and for the data that is produced in the encoded data streams. Another result is the experimental evaluation of the storage overhead. A predicted overhead reduction by compression could be observed in real-data experiments, nonetheless the lower bound of reduction by compression was not reached.

Error correction and erasure codes and steganographic channels use related methods, but are investigated separately. We detail an idea from literature for a steganographic channel in a transmission with error correction code and experimentally investigate it with respect to bandwidth, robustness and detectability. We expand this construction to provide an example of multi-level steganography, i.e., a steganographic channel within a steganographic channel. Furthermore, we investigate the advantages on bandwidth and stealthyness that reversibility of such a steganographic channel brings, together with a new proposal for a covert channel in error-corrected data.

Cauchy-Reed/Solomon coding is applied to tolerate failures of memories and data storage devices in computer systems. In order to obtain a high data access bandwidth, the calculations for coding must be fast and it is required to utilize parallelism. For a software-based system, the most promising approach is data parallelism which can be easily implemented with OpenMP on a multicore or multiprocessor computer. A beneficial aspect is the clear mathematical nature of coding operations that supports functional parallelism as well. We report on a storage system application that generates the encoder and decoder as C-code automatically from a parametric description of the system and inserts OpenMP directives in the code automatically. We compare the performance in terms of achieved data throughput for data parallelism and for functional parallelism that is generated using OpenMP.

Fault-tolerant storage systems tolerate device failures using data redundancy that is brought by erasure-tolerant codes. Devices can fail, can be replaced and data can be recovered with a decoding operation. For classical MDS codes (such as Reed/Solomon codes) this decoding operation is costly and and takes a considerable time because all data elements from a high number of devices must be read and transferred. This high cost is independent on whether a single device failed or many devices failed. Another class of codes that got recently discovered for storage systems - regenerating codes - allow a faster decoding operation for a single device along with a costly recovery operation for multiple failed devices. For a single device repair (regeneration), the cost reduction is reached by shrinking down the amount of data that is transferred. By assumption that a single failure occurs more frequently than multiple concurrent failures, a faster repair is beneficial for the overall reliability of the storage system. Following this motivation, we show how to compare MDS codes and regenerating codes by a reliability model based on Markov chains.

Recently, regenerating codes, a special network coding technique, were discovered for fault-tolerant storage systems with the promising advantage of efficient data recovery in the case of a single node failure and replacement (regeneration case). From the perspective of coding theory, regenerating codes are extensively studied, but there exists no reference on how to implement these codes in storage systems. We provide a comparison of Reed-Solomon codes and regenerating codes from an implementation point of view. The comparison includes the experimental evaluation of the encoding and the regeneration data throughput.

Parallel storage systems distribute data onto several devices. This allows high access bandwidth that is needed for parallel computing systems. It also improves the storage reliability, provided erasure-tolerant coding is applied and the coding is fast enough. In this paper we assume storage systems that apply data distribution and coding in a combined way. We describe, how coding can be done parallel on multicore and GPU systems in order to keep track with the high storage access bandwidth. A framework is introduced that calculates coding equations from parameters and translates them into OpenMP- and OpenCL-based coding modules. These modules do the encoding for data that is written to the storage system, and do the decoding in case of failures of storage devices. We report on the performance of the coding modules and identify factors that influence the coding performance.

Cauchy-Reed/Solomon is an XOR-based erasure-tolerant coding scheme which is widely used for reliable distributed storage and fault-tolerant memory. A variety of different codes can be specified, depending on the number of parallel operating storage resources and the desired strength of fault tolerance. First we present an approach to parameterize the codes for different systems and requirements, such as the desired parallelism and reliability. Based on this parameterization, a Linux block device driver was developed which is evaluated in this paper.

Cauchy Reed/Solomon is an XOR-based erasure-tolerant coding scheme, applied for reliable distributed storage, fault-tolerant memory and reconstruction of content from widely distributed data. The encoding and decoding is based on XOR operations and already well supported by microprocessors.On multicore processors, the coding procedures should also exploit parallelism to speed up coding. In this paper we derive coding procedures from code parameters (e.g. the number of tolerated failures) and propose their transformation into parallel coding schedules that are mapped on multicore processors. We (i) compare functionally decomposed coding procedures with data-parallel coding of different blocks, and (ii) specify the method to derive these schedules.

When systems are designed to tolerate faulty components, application data must be protected against loss. This is reached by a distribution of data together with addition of redundant elements according to an erasure-tolerant code. In this paper, we elaborate architectures for such a fault-tolerant data storage. The concepts are originated from distributed systems and mostly implemented by software. We extend these concepts for usage in the scope of system on chip architectures. On the one hand, systems on chips, and multi core systems are employed as a platform for code calculation - on the other hand, such architectures include these techniques to fulfill their own functionality. We explain how data coding is mapped to (i) multi core CPU structures and (ii) implemented in a specialized design on a FPGA. We compare the time for coding on these architectures for a Cauchy-Reed/Solomon and a classical Reed/Solomon code.

Data distribution together with erasure-tolerant codes allow to store data reliably, even with failed or temporarily disconnected storage resources. The encoding algorithm, i.e. the calculation of the codewords is expressed by XOR equations. Even decoding is the execution of a failure-specific set of equations that are build code-specifically and with knowledge of the failure situation. A new concept for a storage system is to provide encoding equations and decoding equations in advance, as a full description of the code which eliminates the calculations to obtain the recovery strategy. This concept includes that also decoding equations have to be provided in advance, for many different failure situations. This results in a large number of equations and may require a considerable amount of memory, but still a moderate amount - which can be traded for the gained flexibility and simplicity. In this paper we analyze the storage consumption of such a preprocessed decoding equation set. Furthermore, a data structure to access the required equations is proposed. It is shown that codes can be translated into equation sets that are used as parameter set by a storage system.

Building a digital three dimensional representation of a human brain is a challenging task. Such a model provides insights into the microstructure of cortical layering and columns. The presented work is based on a complete dissected and preserved human brain that has been serially sectioned at a coronal resolution that is suitable for single cell detection. More than 6000 sections have been generated and exist as digital images. To obtain a valuable three dimensional representation, morphology preserving affine linear and nonlinear registration schemes are necessary steps. To rebuild a serially sectioned brain, reference images derived from a non deformed object, e.g., MRI or block face images, are necessary for a faithful affine linear and nonlinear registration. In the case of block face images the brain regions must be separated from highly variable background regions to obtain a suitable stack of segmentation images. Among the image segmentation algorithms we found fuzzy c-means techniques as a promising starting point for a sophisticated segmentation framework of either gray level or color images within 2- and 3-dimensions.With respect to algorithmic complexity and computation cost, two fuzzy c-means algorithms were implemented. A proper image preprocessing strategy turned out to be necessary for accurate and robust segmentation results. Primarily, the algorithms work in a parametric resp. supervised mode. Additionally, an automatic mode helps to explore the parameter space within a reasonable range and to compare the segmentation result with an optimal one, provided by an expert. By minimizing the differences we can set up parameters that are used for series of adjacent images. So, it is possible to obtain optimal segmentations independent of illumination disturbances, artifacts and defocusing.We present a complete high resolution and accurate segmentation of the first complete human brain that was sectioned, photographed and digitized at histologic resolution. Based on these images, a succeeding 3D representation is presented. Finally, a segmented and spatially correct straightened data set is available now for coregistration tasks together with the high resolution histologic data set.

In this paper we present an implementation of a Reed/Solomon (R/S) coprocessor to be used on a hybrid computing system, which combines general purpose CPUs with FPGAs. The coprocessor accelerates the encoding of user data to be stored block-wise on a distributed, failure tolerant storage system. We document design constraints and their impact on the resulting architecture. Measurements are presented to characterize the performance of the coprocessor in terms of computation bandwidth, latency, and the hardware-software interaction. For comparison, software based R/S encoding implementations are presented and evaluated as well. Finally, the performance of the hardware accelerated encoding is compared to a software based system.

A hybrid compute system (HCS) combines standard CPUs and reconfigurable devices, usually FPGAs, in one system. These systems have become more attractive again, due to a closer and hence faster coupling of both computational components. From our work with several designs for the same application, we have found the communication between a CPU and a FPGA-based coprocessor to relate either to pipelining or to a bulk-wise transfer with buffered data processing. We identify conditions which determine whether the pipelined or the buffered style should be used in a design. A Reed/Solomon encoding coprocessor has been implemented for each of the communication architectures to serve as an example of how these conditions materialize and how they influence the performance.

Distributed storage systems apply erasure-tolerant codes to guarantee reliable access to data despite failures of storage resources. While many codes can be mapped to XOR operations and efficiently implemented on common microprocessors, only a certain number of codes are usually implemented in a certain system (out of a wide variety of different codes). The ability to include new codes easily, to exchange codes and finally to select codes for several types of data is desirable. To provide this flexibility, a parameterization is used which allows the definition of different XOR based codes, and beyond different styles of en- and decoding. The parameters include (i) the assignment of data and redundancy elements to the storage resources and (ii) a description of en- and decoding algorithms with XOR based equations. The parameters of a certain code can be changed and in addition a wide variety of codes can be described and included in a storage system implementation. The proposed parameterization adopts the ability of codes like EVEN- ODD, Cauchy-R/S and Hover codes to map to distributed resources. Furthermore, en- and decoding algorithms can be described differently, either for minimal coding cost or for minimal coding time on parallel systems.

Distribution of data and erasure tolerant-codes allow to store data reliably in distributed systems. Whereby most techniques are directed to failures of storage resources, also erroneously accessing clients and network interruptions may disturb the storage system operation and cause data loss. Particularly, updates that get partially effective onto distributed data may leave data in an inconsistent state and indirectly destroy data content. Besides, redundancy and data can be left in a state that does not allow to tolerate failures anymore. In this paper, we propose a protocol that takes these issues into account. The protocol forces update consistency in partial distribution scenarios and is correlated with the distribution and coding scheme. It is based on a two-phase commit protocol and a two-layered data structure for buffering updates. For block-wise and sequential access, the protocol cost is hidden in the sequence of accesses related to consecutive blocks.

Distributed storage systems often have to guarantee data availability despite of failures or temporal downtimes of storage nodes. For this purpose, a deletion-tolerant code is applied that allows to reconstruct missing parts in a codeword, i.e. to tolerate a distinct number of failures. The Reed/Solomon (R/S) code is the most general deletion-tolerant code and can be adapted to a required number of tolerable failures. In terms of its least information overhead, R/S is optimal, but it consumes significantly more computation power than parity-based codes. Reconfigurable hardware can be employed for particular operations in finite fields for R/S coding by specialized arithmetics, so that the higher computation effort is compensated by faster and parallel operations. We present architectures for an application-specific acceleration by FPGAs. In this paper, strategies for an efficient communication with the accelerating FPGA and a performance comparison between a pure software-based solution and the accelerated system are provided.

Storing data in distributed systems aims to offer higher bandwidth and scalability than storing locally. But, a couple of disadvantageous issues must be taken into account such as unreliability caused by faults, temporal downtimes and malicious attacks. To improve dependability, redundancy codes like parity can be used as well as more sophisticated codes such as Reed/Solomon. Another issue-security requirements-arise when data is kept in untrusted units in a network. To encrypt data, it is common to use security algorithms like AES. For efficient transfer and storage, the amount of data can be reduced by compression algorithms. All these techniques-data distribution, fault-tolerant coding, encryption and compression-can be employed together using independent algorithms, but in a proper combination. A superposition of these techniques exploiting synergies is still an issue for research. Thus, in this paper we study proper technique combinations applied to distributed storage. The combinations are classified and examined with respect to their potential benefit and limitations. For our model, performance parameters from the distributed storage system NetRAID are used.

Deletion-tolerant codes provide data availability despite storage failures and are commonly used for disk arrays and reliable storage in distributed systems. The codes used for that base on binary parity or on sophisticated cyclic codes with minimal storage overhead. But the calculations for these codes cause either a noticeable number of computation cycles or require a huge number of logic gates. In this paper, a different class of deletion-tolerant codes - (1 out-of N) codes - are analyzed with a focus on their application for distributed storage systems. It is shown that these codes when combined with compression can provide nearly the same low storage overhead as the traditional codes and allow a proper parallelization. Several variants of the code within the design space are discussed. (1 out-of N) codes are compared with parity codes in terms of required time for en- and decoding and their space requirements for parallel logic operations. An integration into a protocol for a distributed storage system is outlined

Reliable distributed data storage systems have to employ redundancy codes to tolerate the loss of storages. Many appropriate codes and algorithms can be found in the literature, but efficient schemes for tolerating several storage failures and their embedding in a distributed system are still research issues. In this paper, a variety of redundancy schemes are compared that got implemented in a distributed storage system. All schemes are based on parity and Reed/Solomon and are integrated in the storage system NetRAID. This system allows to configure several user-specified layouts. A performance and reliability analysis of several data and redundancy layouts is presented that combines analytical and experimental results. In a detail, we present performance results for an optimized Reed/Solomon implementation and give an outline for speeding up encoding and recovery by reconfigurable hardware employed in the distributed storage system