Erasure codes such as Reed-Solomon (RS) codes are being extensively deployed in data centers since they offer significantly higher reliability than data replication methods at much lower storage overheads. These codes however mandate much higher resources with respect to network bandwidth and disk IO during reconstruction of data that is missing or otherwise unavailable. Existing solutions to this problem either demand additional storage space or severely limit the choice of the system parameters. In this paper, we present "Hitchhiker", a new erasure-coded storage system that reduces both network traffic and disk IO by around 25% to 45% during reconstruction of missing or otherwise unavailable data, with no additional storage, the same fault tolerance, and arbitrary flexibility in the choice of parameters, as compared to RS-based systems. Hitchhiker 'rides' on top of RS codes, and is based on novel encoding and decoding techniques that will be presented in this paper. We have implemented Hitchhiker in the Hadoop Distributed File System (HDFS). When evaluating various metrics on the data-warehouse cluster in production at Facebook with real-time traffic and workloads, during reconstruction, we observe a 36% reduction in the computation time and a 32% reduction in the data read time, in addition to the 35% reduction in network traffic and disk IO. Hitchhiker can thus reduce the latency of degraded reads and perform faster recovery from failed or decommissioned machines.

Erasure codes, such as Reed-Solomon (RS) codes, are being increasingly employed in data centers to combat the cost of reliably storing large amounts of data. Although these codes provide optimal storage efficiency, they require significantly high network and disk usage during recovery of missing data. In this paper, we first present a study on the impact of recovery operations of erasure-coded data on the data-center network, based on measurements from Facebook's warehouse cluster in production. To the best of our knowledge, this is the first study of its kind available in the literature. Our study reveals that recovery of RS-coded data results in a significant increase in network traffic, more than a hundred terabytes per day, in a cluster storing multiple petabytes of RS-coded data. To address this issue, we present a new storage code using our recently proposed Piggybacking framework, that reduces the network and disk usage during recovery by 30% in theory, while also being storage optimal and supporting arbitrary design parameters. The implementation of the proposed code in the Hadoop Distributed File System (HDFS) is underway. We use the measurements from the warehouse cluster to show that the proposed code would lead to a reduction of close to fifty terabytes of cross-rack traffic per day.

Facebook recently deployed Facebook Messages, its first ever user-facing application built on the Apache Hadoop platform. Apache HBase is a database-like layer built on Hadoop designed to support billions of messages per day. This paper describes the reasons why Facebook chose Hadoop and HBase over other systems such as Apache Cassandra and Voldemort and discusses the application's requirements for consistency, availability, partition tolerance, data model and scalability. We explore the enhancements made to Hadoop to make it a more effective realtime system, the tradeoffs we made while configuring the system, and how this solution has significant advantages over the sharded MySQL database scheme used in other applications at Facebook and many other web-scale companies. We discuss the motivations behind our design choices, the challenges that we face in day-to-day operations, and future capabilities and improvements still under development. We offer these observations on the deployment as a model for other companies who are contemplating a Hadoop-based solution over traditional sharded RDBMS deployments.

The Hadoop Distributed File System (HDFS) is designed to store very large data sets reliably, and to stream those data sets at high bandwidth to user applications. In a large cluster, thousands of servers both host directly attached storage and execute user application tasks. By distributing storage and computation across many servers, the resource can grow with demand while remaining economical at every size. We describe the architecture of HDFS and report on experience using HDFS to manage 25 petabytes of enterprise data at Yahoo!.

We describe an environment for distributed computing that uses the concept of well-known paradigms. The main advantage of paradigm-oriented distributed computing is that the user only needs to specify application-specific sequential code, while the underlying infrastructure takes care of the parallelization and distribution. The main features of the proposed approach, called PODC, are the following: (1) It is intended for loosely-coupled network environments, not specialized multiprocessors; (2) it is based on an infrastructure of mobile agents; (3) it supports programming in C, rather than a functional or special-purpose language, and (4) it provides a Web-based interactive graphics interface through which programs are constructed, invoked, and monitored. The three paradigms presently supported in PODC are the bag-of-tasks, branch-and-bound search, and genetic programming. We demonstrate their use, implementation, and performance within the mobile agent-based PODC environment.

This paper presents design and experimental results of a parallel linear equation solver by asynchronous partial Gauss-Seidel method. The basic idea of this method is derived from the asynchronous iterative method; newly computed values of unknowns are broadcast to all other processors and are incorporated into computing the next value immediately after they are received. However, since the asynchronous iterative method requires frequent data passing, it is difficult to achieve high performance on practical cluster computing systems due to its enormous communication overhead. To avoid it, the asynchronous partial Gauss-Seidel method reduces frequency of broadcasting new values of unknowns by passing multiple values in a chunk. The experimental results show the advantage of the asynchronous partial Gauss-Seidel method.

Iterative grid-based applications, typified by the finite di fference method for solving partial differential equations [1] and spatially oriented individual-based simulations [2], are important applications for distributed computing. However, iterative grid-based distributed computations requ ir that communication be performed at every step, which can lead to significant communication overhead. We show how a technique called SuperBoundary Exchange (SBX) for significantly reducing this overhead in finite-differen ce computations, first introduced in [3], can be applied to individual-based computations as well, at a cost of a somewhat coarser grid than might otherwise be necessary.

We describe the implementation underlying an environment for distributed computing that uses the concept of well-known paradigms. The main advantage of paradigm oriented distributed computing is that the user only needs to specify application-specific sequential code, while the underlying infrastructure takes care of the parallelization and distribution. The main features of the proposed approach, called PODC, which differentiate it from other approaches, are the following: (1) it is intended for loosely-coupled network environments, not specialized multiprocessors; (2) it is based on an infrastructure of mobile agents; (3) it supports programming in C, rather than a functional or special-purpose language, and (4) it provides a Web based interactive graphics interface through which programs are constructed, invoked, and monitored. The three paradigms presently supported in PODC are the bag-of-tasks, the branch-and-bound and genetic programming. We describe their implementation and performance within the mobile agent based PODC environment.

We introduce a technique for lowering the communication cost in a cer tain type of distributed application, in which processors perform computa tion in each time step and must obtain boundary data from their neighbors before they can perform the next time step. A typical example of such an ap plication is solving differential equations using the finite difference method. Our method, which we call SuperBoundary Exchange, consists of sending a larger boundary less often. This results in less frequent data exchange, trading off against larger messages and some redundant computation. We present experimental data showing that our method consistently results in significant speedup in communication-intensive applications, under varying assumption about the balance of computation load among the processors.

In applications where a large array is distributed over multiple machines, the programming task is simplified if a common global indexing structure is used on all nodes. Distributed Shared Memory (DSM) can be used to achieve this. However, most implementations of DSM are too general and too inefficient for this task. We present a special form of DSM, targeted specifically at numerical distributed applications. This technique decouples the two aspects of DSM systems: the global view and the consistency protocols. The global view allows a program to access each element using its global index even if only a partition of an array is allocated locally. The consistency protocol allows two copies to be synchronized, but leaves the decision of when to synchronize up to the programmer using explicit export/import primitives. The two mechanisms can be used separately or together. The technique provides the advantages of easier programming, flexibility, scalability, and good performance.