Byzantine Fault Tolerant (BFT) consensus forms the foundation of many modern blockchains striving for both high throughput and low latency. A growing bottleneck is transaction execution and validation on the critical path of consensus, which has led to modular decoupled designs that separate ordering from execution: Consensus orders only metadata, while transactions are executed and validated concurrently. While this approach improves performance, it can leave invalid transactions in the ledger, increasing storage costs and enabling new forms of strategic behavior. We present the first systematic study of this setting, providing a formal framework to reason about the interaction between consensus and execution. Using this framework, we show that the decoupled design enables a previously unidentified attack, which we term gaslighting. We prove a fundamental trade-off between resilience to this attack and resource capacity utilization, where both are impossible to achieve deterministically in the decoupled model. To address this trade-off, we discuss an intermediate model for leader-based protocols that is robust to gaslighting attacks while achieving high throughput and low latency.

In Byzantine agreement with predictions each process begins with an input value and some (unreliable) prediction bits. Recently, it has been shown that with classification predictions—where the predictions predict each process to be honest or faulty—Byzantine agreement can be completed more quickly than without predictions, circumventing the traditional Ω(f) round lower bound. However, existing algorithms either handle limited prediction errors or send too many messages. Moreover, they all exchange Ω(n^3) bits—enough to allow the processes to approximately agree on the classifications. In fact, it almost seemed necessary to share a significant number of prediction bits if one wanted to tolerate a high number of incorrect predictions. In this paper, we show that this high level of communication is not inherent to a round-efficient protocol with predictions. We provide an unauthenticated algorithm with near-optimal 𝒪̃(n^2) communication complexity and optimal resilience t < n/3. Furthermore, with authentication, we give an algorithm with optimal 𝒪(n^2κ) communication complexity (where κ is a security parameter) and near-optimal resilience t < (1/2 - ε)n for any constant ε> 0. All of our results have optimal round complexity for any number of errors in the predictions.

In recent years, the explosion of big data and analytics has necessitated distributed storage and processing with several compute nodes (e.g., multiple datacenters). These nodes collaboratively perform parallel computation, where the data is typically partitioned across these nodes to ensure scalability, redundancy and load-balancing. But the nodes may not always be co-located; in many cases, they are part of a larger communication network. Since those nodes only need to communicate among themselves, a key challenge is to design efficient routes catered to that subnetwork. In this work, we initiate the study of distributed sampling and routing problems for subnetworks in any well-connected network. Given any network G = ( V, E) with mixing time tau(mix), consider the canonical problem of permutation routing [Ghaffari, Kuhn and Su, PODC 2017] that aims to minimize both congestion and dilation of the routes, where the demands (i.e., set of source-terminal pairs) are such that each node sends or receives number of messages proportional to its degree. We show that the permutation routing problem, when demands are restricted to any subset S subset of V (i.e., subnetwork), can be solved in exp(O(( root log |S|)) center dot (O) over tilde(tau(mix)) rounds (where (O) over tilde(center dot) hides polylogarithmic factors of |V |). This means that the running time depends subpolynomially on the subnetwork size (i.e., not on the entire network size). The ability to solve permutation routing efficiently immediately implies that a large class of parallel algorithms can be simulated efficiently on the subnetwork. As a prerequisite to constructing efficient routes, we design and analyze distributed branching random walks that distribute tokens started by the nodes in the subnetwork. At a high-level, these algorithms operate by always moving each token according to a (lazy) simple random walk, but also branching a token into multiple tokens at some specified intervals; ultimately, if a node starts a branching walk, with its id in a token, then by the end of execution, several tokens with its id would be randomly distributed among the nodes. As these random walks can be started by many nodes, a crucial challenge is to ensure low-congestion, which is a primary focus of this paper.

Contention resolution addresses the challenge of coordinating access by multiple processes to a shared resource such as memory, disk storage, or a communication channel. Originally spurred by challenges in database systems and bus networks, contention resolution has endured as an important abstraction for resource sharing, despite decades of technological change. Here, we survey the literature on resolving worst-case contention, where the number of processes and the time at which each process may start seeking access to the resource is dictated by an adversary. We also highlight the evolution of contention resolution, where new concerns—such as security, quality of service, and energy efficiency—are motivated by modern systems. These efforts have yielded insights into the limits of randomized and deterministic approaches, as well as the impact of different model assumptions such as global clock synchronization, knowledge of the number of processors, feedback from access attempts, and attacks on the availability of the shared resource.

In the contention-resolution problem, multiple players contend for access to a shared resource. Contention resolution is used in wireless networks, where messages must be transmitted on a shared communication channel. When two or more messages are transmitted at the same time, a collision occurs, and none of the transmissions succeed. Much of the theoretical work on contention resolution has focused on efficiently resolving collisions in order to obtain throughput guarantees. However, in modern-day networks, not all traffic is treated equally. Instead, messages are often handled according to a notion of priority. While throughput remains an important metric, it fails to capture this increasingly-common scenario of traffic prioritization. Motivated by this concern, we design a contention-resolution algorithm where messages have delivery deadlines. Unit-length messages dynamically arrive over time, each with a corresponding delivery deadline that demarcates a window of time wherein the message must be transmitted successfully. We consider inputs that have a feasible schedule, even if message sizes increase by a constant factor. In this setting, we provide an algorithm which guarantees that each message succeeds by its deadline with high probability in its window size.

Contention resolution addresses the problem of coordinating access to a shared communication channel. Time is discretized into synchronized slots, and a packet transmission can be made in any slot. A packet succeeds if it is the only packet transmitted during that slot. If two or more packets are sent in the same slot, then these packets collide and fail. Listening on the channel during a slot provides ternary feedback, indicating whether that slot had (0) silence, (1) a successful transmission, or (2+) noise. No other feedback or exchange of information is available to packets. Packets are (adversarially) injected into the system over time. A packet departs the system once it succeeds. The goal is to ensure all packets succeed, while optimizing throughput, which entails optimizing the fraction of successful slots. Most prior contention resolution algorithms with constant throughput require a short feedback loop, in the sense that a packet's sending probability in slot t+1 is fully determined by its internal state at slot t and the channel feedback at slot t. This paper answers the question of whether these short feedback loops are necessary; that is, how often must listening and updating occur in order to achieve constant throughput? A shared channel can also suffer random or adversarial noise (modeled as jamming), even when no packets are actually sent. How does noise affect our goal of long feedback loops/energy efficiency? Tying these questions together, we ask the following: What does a contention-resolution algorithm have to sacrifice to reduce channel accesses? Must we give up on constant throughput? What about robustness to noise? Here, we show that we need not concede anything by presenting an algorithm with the following guarantees. Suppose there are N packets arriving over time and \scrJ jammed slots, where the input is determined by an adaptive adversary. With high probability in N +\scrJ, our algorithm guarantees \Theta(1) throughput and polylog(N + \scrJ) channel accesses (sends or listens) per packet. We also have analogous guarantees when the input stream is infinite---we prove implicit throughput bounds of \Omega(1) for all time slots t, and this translates to \Theta(1) guaranteed throughput for any slot t where the implicit throughput is sufficiently small in \Theta(1). As a special case, these throughput results give rise to adversarial-queuing theory guarantees.

Byzantine agreement allows n processes to decide on a common value, in spite of arbitrary failures. The seminal Dolev-Reischuk bound states that any deterministic solution to Byzantine agreement exchanges Omega(n^2) bits. In synchronous networks, solutions with optimal O(n^2) bit complexity, optimal fault tolerance, and no cryptography have been established for over three decades. However, these solutions lack robustness under adverse network conditions. Therefore, research has increasingly focused on Byzantine agreement for partially synchronous networks. Numerous solutions have been proposed for the partially synchronous setting. However, these solutions are notoriously hard to prove correct, and the most efficient cryptography-free algorithms still require O(n^3) exchanged bits in the worst case. In this paper, we introduce Oper, the first generic transformation of deterministic Byzantine agreement algorithms from synchrony to partial synchrony. Oper requires no cryptography, is optimally resilient (n >= 3t+1, where t is the maximum number of failures), and preserves the worst-case per-process bit complexity of the transformed synchronous algorithm. Leveraging Oper, we present the first partially synchronous Byzantine agreement algorithm that (1) achieves optimal O(n^2) bit complexity, (2) requires no cryptography, and (3) is optimally resilient (n >= 3t+1), thus showing that the Dolev-Reischuk bound is tight even in partial synchrony. Moreover, we adapt Oper for long values and obtain several new partially synchronous algorithms with improved complexity and weaker (or completely absent) cryptographic assumptions.

Byzantine Agreement (BA) allows.. processes to propose input values to reach consensus on a common, valid L-0 -bit value, even in the presence of up to t < n faulty processes that can deviate arbitrarily from the protocol. Although strategies like randomization, adaptiveness, and batching have been extensively explored to mitigate the inherent limitations of one-shot agreement tasks, there has been limited progress on achieving good amortized performance for multi-shot agreement, despite its obvious relevance to long-lived functionalities such as state machine replication. Observing that a weak form of accountability suffices to identify and exclude malicious processes, we propose new efficient and deterministic multi-shot agreement protocols for multi-value validated Byzantine agreement (MVBA) with a strong unanimity validity property (SMVBA) and interactive consistency (IC). Specifically, let kappa represent the size of the cryptographic objects needed to solve Byzantine agreement when n < 3t. We achieve both IC and SMVBA with.. (1) amortized latency, with a bounded number of slower instances. The SMVBA protocol has O (nL(0) + nk) amortized communication and the IC has O (nL(0) + n(2)k) amortized communication. For input values larger than kappa, our protocols are asymptotically optimal. These results mark a substantial improvement-up to a linear factor, depending on L-0. -over prior results. To the best of our knowledge, the present paper is the first to achieve the long-term goal of implementing a state machine replication abstraction of a distributed service that is just as fast and efficient as its centralized version, but with greater robustness and availability.

We study the problem of Byzantine Agreement with predictions in synchronous message passing systems. Along with a proposal, each process is also given a prediction, i.e., extra information that is not guaranteed to be true. For example, one might imagine that the prediction is produced by a network security monitoring service that looks for patterns of malicious behavior. Our goal is to design an algorithm that is more efficient when the predictions are accurate, smoothly degrades in performance as predictions decrease in accuracy, and in the worst case performs (almost) as well as any algorithm without predictions even when the predictions are completely inaccurate. On the negative side, we show that Byzantine Agreement with predictions still requires Omega(n + t(2)) messages, even in executions where the predictions are completely accurate. On the positive side, we show that classification predictions, which provide information about which processes might be faulty, can help improve the round complexity of synchronous Byzantine Agreement. We present new algorithms that leverage classification predictions to yield better round complexity, and we show that the round complexity achieved is (almost) optimal as a function of the prediction quality.

Blockchains have become ubiquitous in the world of robust decentralized applications. A crucial requirement for implementing a blockchain is an “overlay network” providing reliable and efficient communication among the participants. In this work, we provide communication-efficient and churn-optimal (barring log factors) Byzantine-resilient algorithms for maintaining blockchain networks. Our approach utilizes an interesting “cross-layer optimization” wherein the overlay network relies on the blockchain that is built on top of it. An important contribution is a tight “half-life” analysis on the amount of churn that can be tolerated, where peers have bandwidth restrictions. Moreover, by leveraging synergies between the blockchain and the overlay, we provide non-trivial recovery guarantees from catastrophic failures, which include a large class of connectivity issues such as denial-of-service, or exponentially unlikely lucky streaks for Byzantine peers, etc.

Byzantine Agreement (BA) enables n processes to reach consensus on a common valid L o -bit value, even in the presence of up to t < n faulty processes that can deviate arbitrarily from their prescribed protocol. Despite its significance, the optimal communication complexity for key variations of BA has not been determined within the honest majority regime ( n = 2 t +1), for both the worst-case scenario and the adaptive scenario, which accounts for the actual number f ≤ t of failures. We introduce ada-Dare (Adaptively Disperse, Agree, Retrieve), a novel universal approach to solve BA efficiently. Let κ represent the size of the cryptographic objects required to solve BA when t > n /3. Different instantiations of ada-Dare achieve near-optimal adaptive bit complexity of O ( nL o + n ( f + 1) κ ) for both strong multi-valued validated BA (SMVBA) and interactive consistency (IC). By definition, for IC, L o = nL in where L in is the size of an input value. These results achieve optimal O ( n ( L o + f )) word complexity and significantly improve the previous best results by up to a linear factor, depending on L o and f.

Design of an efficient thread-safe concurrent data structure is a balancing act between its implementation complexity and performance. Lock-based concurrent data structures, which are relatively easy to derive from their sequential counterparts and to prove thread-safe, suffer from poor throughput under even light multi-threaded workload. At the same time, lock-free concurrent structures allow for high throughput, but are notoriously difficult to get right and require careful reasoning to formally establish their correctness. We explore a solution to this conundrum based on batch parallelism, an approach for designing concurrent data structures via a simple insight: efficiently processing a batch of a priori known operations in parallel is easier than optimising performance for a stream of arbitrary asynchronous requests. Alas, batch-parallel structures have not seen wide practical adoption due to (i) the inconvenience of having to structure multi-threaded programs to explicitly group operations and (ii) the lack of a systematic methodology to implement batch-parallel structures as simply as lock-based ones. We present OBatcher-an OCaml library that streamlines the design, implementation, and usage of batch-parallel structures. It solves the first challenge (how to use) by suggesting a new lightweight implicit batching design that is built on top of generic asynchronous programming mechanisms. The second challenge (how to implement) is addressed by identifying a family of strategies for converting common sequential structures into efficient batch-parallel ones. We showcase OBatcher with a diverse set of benchmarks. Our evaluation of all the implementations on large asynchronous workloads shows that (a) they consistently outperform the corresponding coarse-grained lock-based implementations and that (b) their throughput scales reasonably with the number of processors.

The best known solutions for k -message broadcast in dynamic networks of size n require Ω ( nk ) rounds. In this article, we see if these bounds can be improved by smoothed analysis. To do so, we study perhaps the most natural randomized algorithm for disseminating tokens in this setting: at every timestep, choose a token to broadcast randomly from the set of tokens you know. We show that with even a small amount of smoothing (i.e., one random edge added per round), this natural strategy solves k -message broadcast in \(\tilde{O}(n+k^3)\) rounds, with high probability, beating the best known bounds for \(k=o(\sqrt {n})\) and matching the Ω ( n + k ) lower bound for static networks for k = O ( n 1/3 ) (ignoring logarithmic factors). In fact, the main result we show is even stronger and more general: Given ℓ-smoothing (i.e., ℓ random edges added per round), this simple strategy terminates in \(O(kn^{2/3}\log ^{1/3}(n)\ell ^{-1/3})\) rounds. We then prove this analysis close to tight with an almost-matching lower bound. To better understand the impact of smoothing on information spreading, we next turn our attention to static networks, proving a tight bound of \(\tilde{O}(k\sqrt {n})\) rounds to solve k -message broadcast, which is better than what our strategy can achieve in the dynamic setting. This confirms the intuition that although smoothed analysis reduces the difficulties induced by changing graph structures, it does not eliminate them altogether. Finally, we apply tools developed to support our smoothed analysis to prove an optimal result for k -message broadcast in so-called well-mixed networks in the absence of smoothing. By comparing this result to an existing lower bound for well-mixed networks, we establish a formal separation between oblivious and strongly adaptive adversaries with respect to well-mixed token spreading, partially resolving an open question on the impact of adversary strength on the k -message broadcast problem.

Byzantine agreement enables n processes to agree on a common L-bit value, despite up to t > 0 arbitrary failures. A long line of work has been dedicated to improving the bit complexity of Byzantine agreement in synchrony. This has culminated in COOL, an error-free (deterministically secure against a computationally unbounded adversary) solution that achieves O( nL + n(2) log n) worst-case bit complexity (which is optimal for L >= n log n according to the Dolev-Reischuk lower bound). COOL satisfies strong unanimity: if all correct processes propose the same value, only that value can be decided. Whenever correct processes do not agree a priori (there is no unanimity), they may decide a default value perpendicular to from COOL. Strong unanimity is, however, not sufficient for today's state machine replication (SMR) and blockchain protocols. These systems value progress and require a decided value to always be valid (according to a predetermined predicate), excluding default decisions (such as perpendicular to) even in cases where there is no unanimity a priori. Validated Byzantine agreement satisfies this property (called external validity). Yet, the best error-free (or even signature-free) validated agreement solutions achieve only O(n(2) L) bit complexity, a far cry from the Omega(nL + n(2)) Dolev-Reischuk lower bound. Is it possible to bridge this complexity gap? We answer the question affirmatively. Namely, we present two new synchronous algorithms for validated Byzantine agreement, HASHEXT and ERRORFREEEXT, with different trade-offs. Both algorithms are (1) signature-free, (2) optimally resilient (tolerate up to t < n/3 failures), and (3) early-stopping (terminate in O( f + 1) rounds, where f <= t denotes the actual number of failures). On the one hand, HASHEXT uses only hashes and achieves O( nL + n(3)kappa) bit complexity, which is optimal for L >= n(2)kappa (where kappa is the size of a hash). On the other hand, ERRORFREEEXT is error-free, using no cryptography whatsoever, and achieves O (nL+ n(2)) log n) bit complexity, which is near-optimal for any L.

Byzantine agreement enables n processes to agree on a common L-bit value, despite t > 0 arbitrary failures. A long line of work has been dedicated to improving the worst-case bit complexity of Byzantine agreement in synchrony. This has culminated in COOL, an error-free (deterministically secure against a computationally unbounded adversary) algorithm that achieves a near-optimal bit complexity of O(nL + n^2 log n). COOL satisfies strong validity: if all correct processes propose the same value, only that value can be decided. Thus, whenever correct processes do not a priori agree, COOL might decide on "bottom", thus limiting its application in today's state machine replication (SMR) and blockchain protocols. In this work, we focus on the aforementioned limitation. Can we design an error-free near-optimal Byzantine agreement algorithm applicable in today's SMR and blockchain protocols? Can we design an error-free near-optimal agreement algorithm with external validity (a.k.a. validated agreement) stipulating that only values valid according to a predetermined predicate can be decided? This paper answers the question affirmatively. Namely, we present EXT, an error-free synchronous Byzantine agreement algorithm that satisfies external (along with strong) validity while exchanging O(n log n L + n^2 log n) bits in the worst case. Importantly, EXT is optimally resilient (tolerates t < n / 3 failures) and terminates in optimal O(n) rounds. Perhaps surprisingly, we construct EXT by exploiting existing concepts: (1) the recursive framework proposed by Berman, Garay and Perry and Coan and Welch and recently restated by Momose and Ren, (2) the aforementioned COOL algorithm introduced by Chen, and (3) the data dissemination primitive introduced by Das, Xiang and Ren.

Contention resolution addresses the problem of coordinating access to a shared channel. Time proceeds in slots, and a packet transmission can be made in any slot. A packet is successfully sent if no other packet is also transmitted during that slot. If two or more packets are sent in the same slot, then none of these transmissions succeed. Listening during a slot gives ternary feedback, indicating if that slot had (0) silence, (1) a successful transmission, or (2+) noise. No other feedback is available. Packets are (adversarially) injected into the system over time. A packet departs the system once it is successful. The goal is to send all packets while optimizing throughput, which is roughly the fraction of successful slots. Most prior algorithms with constant throughput require a short feedback loop, in the sense that a packet's sending probability in slot t+1 is fully determined by its internal state at slot t and the channel feedback at slot t. An open question is whether these short feedback loops are necessary; that is, how often must listening and updating occur in order to achieve constant throughput? This question addresses energy efficiency, since both listening and sending consume significant energy. The channel can also suffer adversarial noise ("jamming"), which causes any listener to hear noise, even when no packets are sent. How does jamming affect our goal of long feedback loops/energy efficiency? Connecting these questions, we ask: what does a contention-resolution algorithm have to sacrifice to reduce channel accesses? Must we give up on constant throughput or robustness to noise? Here, we show that we need not concede anything. Suppose there are N packets and J jammed slots, where the input is determined by an adaptive adversary. We give an algorithm that, with high probability in N+J, has constant throughput and polylog(N+J) channel accesses per packet.

Byzantine consensus allows n processes to decide on a common value, in spite of arbitrary failures. The seminal Dolev-Reischuk bound states that any deterministic solution to Byzantine consensus exchanges Omega(n^2) bits. In recent years, great advances have been made in deterministic Byzantine agreement for partially synchronous networks, with state-of-the-art cryptographic solutions achieving O(n^2 \kappa) bits (where $\kappa$ is the security parameter) and nearly matching the lower bound. In contrast, for synchronous networks, optimal solutions with O(n^2) bits, with no cryptography and the same failure tolerance, have been known for more than three decades. Can this gap in network models be closed? In this paper, we present Repeater, the first generic transformation of Byzantine agreement algorithms from synchrony to partial synchrony. Repeater is modular, relying on existing and novel algorithms for its sub-modules. With the right choice of modules, Repeater requires no additional cryptography, is optimally resilient (n = 3t+1, where t is the maximum number of failures) and, for constant-size inputs, preserves the worst-case per-process bit complexity of the transformed synchronous algorithm. Leveraging Repeater, we present the first partially synchronous algorithm that (1) achieves optimal bit complexity (O(n^2) bits), (2) resists a computationally unbounded adversary (no cryptography), and (3) is optimally-resilient (n = 3t+1), thus showing that the Dolev-Reischuk bound is tight in partial synchrony. Moreover, we adapt Repeater for long inputs, introducing several new algorithms with improved complexity and weaker (or completely absent) cryptographic assumptions.

Byzantine Fault-Tolerant (BFT) protocols are known to be difficult to design and to reason about. To address this challenge, on one hand, several approaches have been developed recently for computer-aided formal verification of the desired correctness properties, both safety and liveness, of standalone BFT protocols. On the other hand, the distributed computing community has made attempts to reduce the conceptual complexity of constructing new such protocols by showing how to assemble them from simpler "building blocks". No methodology to date combines these two approaches for foundational verification of arbitrary BFT protocols. We present Bythos, the first foundational framework for compositional mechanised verification of both safety and liveness of composite BFT protocols. Bythos is implemented on top of the Coq proof assistant and uses Coq's higher-order logic to reuse proofs of common facts about knowledge and trust in BFT protocols. It allows for compact liveness specifications in the style of TLA+, and for their proofs using an embedding of TLA into Coq. Most importantly, Bythos provides a family of higher-order definitions that allow building composite BFT protocols from simpler ones, with their correctness proofs derived. We showcase Bythos by verifying in it safety and liveness properties of three basic BFT protocols: Reliable Broadcast, Provable Broadcast, and the recently proposed Accountable Byzantine Confirmer, as well as their compositions.

The strong Byzantine agreement (SBA) problem is defined among n processes, out of which t < n can be faulty and behave arbitrarily. SBA allows correct (non-faulty) processes to agree on a common value. Moreover, if all correct processes have proposed the same value, only that value can be agreed upon. It has been known for a long time that any solution to the SBA problem incurs quadratic worst-case word complexity; additionally, the bound was known to be tight. However, no existing protocol achieves adaptive word complexity, where the number of exchanged words depends on the actual number of faults, and not on the upper bound. Therefore, it is still unknown whether SBA with adaptive word complexity exists. This paper answers the question in the affirmative. Namely, we introduce STRONG, a synchronous protocol that solves SBA among n = (2 + Omega(1))t + 1 processes and achieves adaptive word complexity. We show that the fundamental challenge of adaptive SBA lies in efficiently solving certification, the problem of obtaining a constant-sized, locally-verifiable proof that a value can safely be decided.