Wormhole is a publish-subscribe (pub-sub) system developed for use within Facebook's geographically replicated datacenters. It is used to reliably replicate changes among several Facebook services including TAO, Graph Search and Memcache. This paper describes the design and implementation of Wormhole as well as the operational challenges of scaling the system to support the multiple data storage systems deployed at Facebook. Our production deployment of Wormhole transfers over 35 GBytes/sec in steady state (50 millions messages/sec or 5 trillion messages/day) across all deployments with bursts up to 200 GBytes/sec during failure recovery. We demonstrate that Wormhole publishes updates with low latency to subscribers that can fail or consume updates at varying rates, without compromising efficiency.

Memcached is a well known, simple, in-memory caching solution. This paper describes how Facebook leverages memcached as a building block to construct and scale a distributed key-value store that supports the world's largest social network. Our system handles billions of requests per second and holds trillions of items to deliver a rich experience for over a billion users around the world.

We introduce a simple data model and API tailored for serving the social graph, and TAO, an implementation of this model. TAO is a geographically distributed data store that provides efficient and timely access to the social graph for Facebook's demanding workload using a fixed set of queries. It is deployed at Facebook, replacing memcache for many data types that fit its model. The system runs on thousands of machines, is widely distributed, and provides access to many petabytes of data. TAO can process a billion reads and millions of writes each second.

This paper examines the workload of Facebook's photo-serving stack and the effectiveness of the many layers of caching it employs. Facebook's image-management infrastructure is complex and geographically distributed. It includes browser caches on end-user systems, Edge Caches at ~20 PoPs, an Origin Cache, and for some kinds of images, additional caching via Akamai. The underlying image storage layer is widely distributed, and includes multiple data centers. We instrumented every Facebook-controlled layer of the stack and sampled the resulting event stream to obtain traces covering over 77 million requests for more than 1 million unique photos. This permits us to study traffic patterns, cache access patterns, geolocation of clients and servers, and to explore correlation between properties of the content and accesses. Our results (1) quantify the overall traffic percentages served by different layers: 65.5% browser cache, 20.0% Edge Cache, 4.6% Origin Cache, and 9.9% Backend storage, (2) reveal that a significant portion of photo requests are routed to remote PoPs and data centers as a consequence both of load-balancing and peering policy, (3) demonstrate the potential performance benefits of coordinating Edge Caches and adopting S4LRU eviction algorithms at both Edge and Origin layers, and (4) show that the popularity of photos is highly dependent on content age and conditionally dependent on the social-networking metrics we considered.

This paper describes Haystack, an object storage system optimized for Facebook's Photos application. Facebook currently stores over 260 billion images, which translates to over 20 petabytes of data. Users upload one billion new photos (∼60 terabytes) each week and Facebook serves over one million images per second at peak. Haystack provides a less expensive and higher performing solution than our previous approach, which leveraged network attached storage appliances over NFS. Our key observation is that this traditional design incurs an excessive number of disk operations because of metadata lookups. We carefully reduce this per photo metadata so that Haystack storage machines can perform all metadata lookups in main memory. This choice conserves disk operations for reading actual data and thus increases overall throughput.

Integrated microbatteries are being currently developed to act as a "micropower" source in microsatellites. They help provide localized current capacities or embedded power supplies at the chip level, for space exploration. These power cells are designed to be rechargeable. This research paper aims at presenting charging these power cells using pulsing algorithms developed at MRCI with an on chip pulse charger controller.

We present two OBDD based model checking algorithms for the verification of Nash equilibria in finite state mechanisms modeling Multiple Administrative Domains (MAD) distributed systems with possibly colluding agents (coalitions) and with possibly faulty or malicious nodes (Byzantine agents). Given a finite state mechanism, a proposed protocol for each agent and the maximum sizes f for Byzantine agents and q for agents collusions, our model checkers return Pass if the proposed protocol is an ε-f-q-Nash equilibrium, i.e. no coalition of size up to q may have an interest greater than ε in deviating from the proposed protocol when up to f Byzantine agents are present, Fail otherwise. We implemented our model checking algorithms within the NuSMV model checker: the first one explicitly checks equilibria for each coalition, while the second represents symbolically all coalitions. We present experimental results showing their effectiveness for moderate size mechanisms. For example, we can verify coalition Nash equilibria for mechanisms which corresponding normal form games would have more than 5 ×1021 entries. Moreover, we compare the two approaches, and the explicit algorithm turns out to outperform the symbolic one. To the best of our knowledge, no model checking algorithm for verification of Nash equilibria of mechanisms with coalitions has been previously published.

Cooperative services are an increasingly popular way to deploy applications. In these services, computers controlled by different entities are supposed to work together to achieve a common goal, such as file-sharing [6, 12], mesh routing [2], and peer-to-peer backup [1, 7]. The resulting systems can be more fault-tolerant, scalable, and less expensive than traditional client-server approaches. However, a cooperative service may never see these benefits if it does not tolerate Byzantine users who may disrupt the service or selfish ones who may try to use it without contributing their fair share. One of the key challenges in designing a cooperative service is ensuring that participants actually cooperate. Several deployed applications [6, 12] and research prototypes [7, 17] include incentives and punishments to encourage cooperation. However, selfish participants still frequently find ways to cheat. For example, in the KaZaA [12] network almost half the users falsify their contributions by using a hacked binary [13]. In BitTorrent [6], researchers have found ways to free-ride [19]. These vulnerabilities are symptoms of a deeper problem: almost no cooperative service rigorously shows that its incentives and punishments are sufficient to induce cooperation. In my research, I aim to build cooperative systems that tolerate the actions of Byzantine users while removing incentives for selfish users to cheat. In this adversarial environment, constructing systems that work well and are robust requires me to step back and forth between practice and theory, something that distinguishes my research from that of many others who design these systems. I believe that pursuing the practice-theory cycle—letting practice guide theory and vice versa—is necessary in building robust and practical systems. This approach requires me to be persistent, resourceful, and creative. Drawing upon these traits and my training so far, I plan to i) develop the theoretical framework necessary to reason about selfish and Byzantine behaviors in cooperative services, ii) design efficient and scalable mechanisms in this framework to address practical issues in building real services, and iii) implement large-scale prototypes that are competitive with if not better than existing less robust works.

Byzantine and rational behaviors are increasingly recognized as unavoidable realities in todaypsilas cooperative services. Yet, how to design BAR-tolerant protocols and rigorously prove them strategy proof remains somewhat of a mystery: existing examples tend either to focus on unrealistically simple problems or to want in rigor. The goal of this paper is to demystify the process by presenting the full algorithmic development cycle that, starting from the classic synchronous repeated terminating reliable broadcast (R-TRB) problem statement, leads to a provably BAR-tolerant solution. We show i) how to express R-TRB as a game; ii) why the strategy corresponding to the optimal Byzantine fault tolerant algorithm of Dolev and strong does not guarantee safety when non-Byzantine players behave rationally; iii) how to derive a BAR-tolerant R-TRB protocol: iv) how to prove rigorously that the protocol ensures safety in the presence of non-Byzantine rational players.

We present FlightPath, a novel peer-to-peer streaming application that provides a highly reliable data stream to a dynamic set of peers. We demonstrate that FlightPath reduces jitter compared to previous works by several orders of magnitude. Furthermore, FlightPath uses a number of run-time adaptations to maintain low jitter despite 10% of the population behaving maliciously and the remaining peers acting selfishly. At the core of FlightPath's success are approximate equilibria. These equilibria allow us to design incentives to limit selfish behavior rigorously, yet they provide sufficient flexibility to build practical systems. We show how to use an Ɛ-Nash equilibrium, instead of a strict Nash, to engineer a live streaming system that uses bandwidth efficiently, absorbs flash crowds, adapts to sudden peer departures, handles churn, and tolerates malicious activity.

Integrated circuits have been invented for managing the charging and discharging of such advanced miniature energy-storage devices as planar arrays of microscopic energy-storage elements [typically, microscopic electrochemical cells (microbatteries) or microcapacitors]. The architecture of these circuits enables implementation of the following energy-management options: dynamic configuration of the elements of an array into a series or parallel combination of banks (subarrarys), each array comprising a series of parallel combination of elements; direct addressing of individual banks for charging/or discharging; and, disconnection of defective elements and corresponding reconfiguration of the rest of the array to utilize the remaining functional elements to obtain the desited voltage and current performance. An integrated circuit according to the invention consists partly of a planar array of field-effect transistors that function as switches for routing electric power among the energy-storage elements, the power source, and the load. To connect the energy-storage elements to the power source for charging, a specific subset of switches is closed; to connect the energy-storage elements to the load for discharging, a different specific set of switches is closed. Also included in the integrated circuit is circuitry for monitoring and controlling charging and discharging. The control and monitoring circuitry, the switching transistors, and interconnecting metal lines are laid out on the integrated-circuit chip in a pattern that registers with the array of energy-storage elements. There is a design option to either (1) fabricate the energy-storage elements in the corresponding locations on, and as an integral part of, this integrated circuit; or (2) following a flip-chip approach, fabricate the array of energy-storage elements on a separate integrated-circuit chip and then align and bond the two chips together.

Gossip-based communication protocols are often touted as being robust. Not surprisingly, such a claim relies on assumptions under which gossip protocols are supposed to operate. In this paper, we discuss and in some cases expose some of these assumptions and discuss how sensitive the robustness of gossip is to these assumptions. This analysis gives rise to a collection of new research challenges.

We introduce the Paxos register to simplify and unify the presentation of Paxos-style consensusprotocols. We use our register to show how Lamport7.s Classic Paxos and Castro and Liskov's Byzantine Paxos are the same consensusprotocol, but for different failure models. We also use our register to compare and contrast Byzantine Paxos with Martin and Alvisi's Fast Byzantine Consensus. The Paxos register is a write-once register that exposes two important abstractions for reaching consensus: ( i ) read and write operations that capture how processes in Pams protocols progose and decide values and (ii) tokens that capture how these protocols guarantee agreement despite partial failures. We encapsulate the difSerences of several Paxos-style protocols in the implementation details of these abstractions.

Most proposed gossip-based systems use an ad-hoc design. We observe a low degree of reutilization among this proposals. We present how this limits both the systematic development of gossip-based applications and the number of applications that can benefit from gossip-based construction. We posit that these reinvent-the-wheel approaches poses a significant barrier to the spread and usability of gossip protocols. This paper advocates a conceptual design framework based upon aggregating basic and predefined building blocks (B2). We show how to compose building blocks within our framework to construct more complex blocks to be used in gossip-based applications. The concept is further depicted with two gossip-based applications described using our building blocks.

Distributed systems that span multiple administrative domains require protocols that tolerate both Byzantine and selfish nodes. This paper offers a theory that can be used to analyze such protocols. The theory systematically extends traditional game theory solution concepts through an ex ante analysis that incorporates a rational player’s awareness of the possible presence of Byzantine players in the player’s utility function. We illustrate our approach by modeling synchronous Terminating Reliable Broadcast as a game. We show that Dolev and Strong’s Byzantine TRB protocol with message authentication is not a Nash equilibrium and that rational deviations from it may lead to violation of the TRB safety properties. We present a new TRB protocol with the same asymptotic complexity of Dolev-Strong and prove it to be a Nash equilibrium. Finally, we prove that (k-t) robustness, a recently proposed solution concept for games with Byzantine and rational players, cannot yield an equilibrium in games, such as our TRB game, that model systems where any node may crash and communication is necessary and incurs cost.

We present Information Theoretically secure Byzantine Paxos (IT ByzPaxos), the first deterministic asynchronous Byzantine consensus protocol that is provably secure despite a computationally unbounded adversary. Previous deterministic asynchronous algorithms for Byzantine consensus rely on unproven number theoretic assumptions (i.e., digital signatures) to maintain agreement. IT ByzPaxos instead uses secret sharing techniques that are information theoretically secure to ensure that all correct processes agree. Our protocol guarantees safety in an asynchronous system and provides progress under eventual synchrony. IT ByzPaxos matches the 3f+1 lower bound on the number of processes for Byzantine consensus.