Most databases can be configured to operate under isolation levels weaker than serializability. These enforce fewer restrictions on the concurrent access to data and consequently allow for more performant implementations. While formal frameworks for rigorously specifying isolation levels exist, reasoning about the semantic differences between specifications remains notoriously difficult. This paper proposes a tool – Isolde – that can automatically generate examples that are allowed by an isolation level but disallowed by another. This simple primitive unlocks a range of useful reasoning tasks, including checking equivalence between definitions, and verifying (by refutation) subtle semantic properties of isolation levels. For example, Isolde allowed us to easily and automatically reproduce a famously elusive result from the literature and to discover a previously unknown bug in the alternative specification of a standard isolation level used in a state-of-the-art isolation checker.

Machine learning models are growing, with some large language models reaching a scale of billions of trainable parameters. Training these models has since become one of the most data-hungry and computation-heavy tasks. Efforts to distribute the training task mostly follow a federated approach, where a central server oversees the training process. This approach: 1) raises concerns about data privacy; and 2) creates a single point of failure. Current proposals for a fully decentralized approach often rely on costly broadcasts to disseminate model updates and do not tolerate heterogeneity in the training data, as it makes detecting Byzantine contributions harder. We propose BLADE, a generalized fully decentralized (and asynchronous) Byzantine fault-tolerant machine learning algorithm. BLADE was designed to be configurable and adapt to harsh environments, and significantly reduces the communication overhead compared to the state of the art. We performed a comprehensive empirical evaluation, and results confirm models trained with BLADE can achieve an accuracy comparable to a centralized training instance, even if the data distribution among peers is heterogeneous, and robustly aggregate model updates in the presence of Byzantine attacks, and even against sporadic Byzantine majorities.

The Raft agreement algorithm is recognized for its ease of understanding and practical implementation, and is currently adopted in systems such as Kubernetes. However, it has some limitations in terms of scalability and performance as it concentrates effort on the leader. In this paper we present a new algorithm that expands Raft by incorporating epidemic propagation mechanisms to decentralize the replication effort. Our proposal is evaluated experimentally with a Go implementation and tested with a significant number of processes. – – O algoritmo de acordo Raft é reconhecido pela sua facilidade de compreensão e implementação prática, sendo atualmente adotado em sistemas como o Kubernetes. No entanto, tem algumas limitações em termos de escalabilidade e desempenho por concentrar o esforço no líder. Neste trabalho apresentamos um novo algoritmo que expande o Raft com a incorporação de mecanismos de propagação epidémica para descentralizar o esforço da replicação. A nossa proposta é avaliada experimentalmente com uma implementação em Go e testada com um número significativo de processos.

Developers of data-intensive georeplicated applications face a difficult decision when selecting a database system. As captured by the CAP theorem, CP systems such as Spanner provide strong consistency that greatly simplifies application development. AP systems such as AntidoteDB providing Transactional Causal Consistency (TCC), ensure availability in face of network partitions and isolate performance from wide-area round-trip times, but avoid lost-update anomalies only when values can be merged. Ideally, an application should be able to adapt to current data and network conditions by selecting which transactional consistency to use for each transaction. In this paper, we test the hypothesis that a georeplicated database system can be built at its core providing only TCC, hence, being AP, but allow an application to execute some transactions under Snapshot Isolation (SI), hence CP. Our main result is showing that this can be achieved even when all the interaction happens through the TCC database system, without additional communication channels between the participants. A preliminary experimental evaluation with a proof-of-concept implementation using AntidoteDB shows that this approach is feasible.

Diversity is crucial in systems that tolerate Byzantine faults. Traditionally, system builders have relied on standardized interfaces (e.g., POSIX for operating systems) to obtain off-the-shelf components or on n-version programming for custom functionality. Unfortunately, standardized alternatives are rare, and the independent development of multiple versions of the same software is costly and justified only on the most critical applications. In this paper, we show that a limited and focused use of LLMs for translation opens up the possibility of leveraging the existing diversity in functionally equivalent but non-standardized components. Specifically, we show that LLMs can produce functionally correct database query translations with minimal guidance and adapt to diverse data models and query contexts, enabling the use of radically different database models, both SQL and NoSQL, together in a Byzantine fault-tolerant replicated system. We outline an approach to achieve this in practice and discuss future research directions.

Data intensive applications increasingly make use of multiple data stores in the cloud, providing a diversity of data and query models, as well as durability and scale trade-offs. However, this has a severe impact on reliability, as the key fault-tolerance mechanism for database systems, i.e. ACID transactions, is no longer available. Although it is possible to implement transactions without changes to the database servers, this either requires a proxy server, which compromises scale and availability, or a client-side layer that changes the data schema, excludes legacy applications, and adds significant overhead. We address this challenge with a proposal to delegate functionality from a client-side transactional layer to a server-side query engine such that compatibility with legacy applications is restored. We implemented a proof-of-concept and show that it significantly improves performance for analytical applications.

There are now multiple proposals for Conflict-free Replicated Data Types (CRDTs) in SQL databases aimed at distributed systems. Some, such as ElectricSQL, provide only relational tables as convergent replicated maps, but this omits semantics that would be useful for merging updates. Others, such as Pg\_crdt, provide access to a rich library of encapsulated column types. However, this puts merge and query processing outside the scope of the query optimizer and restricts the ability of an administrator to influence access paths with materialization and indexes. Our proposal, CRDV, overcomes this challenge by using two layers implemented as SQL views: The first provides a replicated relational table from an update history, while the second implements varied and rich types on top of the replicated table. This allows the definition of merge semantics, or even entire new data types, in SQL itself, and enables global optimization of user queries together with merge operations. Therefore, it naturally extends the scope of query optimization and local transactions to operations on replicated data, can be used to reproduce the functionality of common CRDTs with simple SQL idioms, and results in better performance than alternatives.

We present LazyFS, a new fault injection tool that simplifies the debugging and reproduction of complex data durability bugs experienced by databases, key-value stores, and other data-centric systems in crashes. Our tool simulates persistence properties of POSIX file systems (e.g., operations ordering and atomicity) and enables users to inject lost and torn write faults with a precise and controlled approach. Further, it provides profiling information about the system's operations flow and persisted data, enabling users to better understand the root cause of errors. We use LazyFS to study seven important systems: PostgreSQL, etcd, Zookeeper, Redis, LevelDB, PebblesDB, and Lightning Network. Our fault injection campaign shows that LazyFS automates and facilitates the reproduction of five known bug reports containing manual and complex reproducibility steps. Further, it aids in understanding and reproducing seven ambiguous bugs reported by users. Finally, LazyFS is used to find eight new bugs, which lead to data loss, corruption, and unavailability.

Modern supercomputers host numerous jobs that compete for shared storage resources, causing I/O interference and performance degradation. Solutions based on software- defined storage (SDS) emerged to address this issue by coordinating the storage environment through the enforcement of QoS policies. However, these often fail to consider the scale of modern HPC infrastructures.In this work, we explore the advantages and shortcomings of state-of-the-art SDS solutions and highlight the scale of current production clusters and their rising trends. Furthermore, we conduct the first experimental study that sheds new insights into the performance and scalability of flat and hierarchical SDS control plane designs.Our results, using the Frontera supercomputer, show that a flat design with a single controller can scale up to 2,500 nodes with an average control cycle latency of 41 ms, while hierarchical designs can handle up to 10,000 nodes with an average latency ranging between 69 and 103 ms.

While a significant number of databases are deployed in cloud environments, pushing part or all data storage and querying planes closer to their sources (i.e., to the edge) can provide advantages in latency, connectivity, privacy, energy and scalability. This article dissects the advantages provided by databases in edge and fog environments, by surveying application domains and discussing the key drivers for pushing database systems to the edge. At the same time, it also identifies the main challenges faced by developers in this new environment, and analysis the mechanisms employed to deal with them. By providing an overview of the current state of edge and fog databases, this survey provides valuable insights into future research directions.

ThereisagrowingdemandforpersistentdatainIoT,edgeandsimilarresource-constraineddevices. However, standardFLASH memory-based solutions present performance, energy, and reliability limitations in these applications. We propose MRAM persistent memory as an alternative to FLASH based storage. Preliminary experimental results show that its performance, power consumption, and reliability in typical database workloads is competitive for resource-constrained devices. This opens up new opportunities, as well as challenges, for small-scale database systems. MRAM is tested for its raw performance and applicability to key-value and relational database systems on resource-constrained devices. Improvements of as much as three orders of magnitude in write performance for key-value systems were observed in comparison to an alternative NAND FLASH based device.

Transactions have been a key issue in database management for a long time and there are a plethora of architectures and algorithms to support and implement them. The current state-of-the-art is focused on storage management and is tightly coupled with its design, leading, for instance, to the need for completely new engines to support new features such as Hybrid Transactional Analytical Processing (HTAP). We address this challenge with a proposal to implement transactional logic in a query language such as SQL. This means that our approach can be layered on existing analytical systems but that the retrieval of a transactional snapshot and the validation of update transactions runs in the server and can take advantage of advanced query execution capabilities of an optimizing query engine. We demonstrate our proposal, TiQuE, on MonetDB and obtain an average 500x improvement in transactional throughput while retaining good performance on analytical queries, making it competitive with the state-of-the-art HTAP systems.

Automatic grading based on unit tests is a key feature of massive open online courses (MOOC) on programming, as it allows instant feedback to students and enables courses to scale up. This technique works well for sequential programs, by checking outputs against a sample of inputs, but unfortunately it is not adequate for detecting races and deadlocks, which precludes its use for concurrent programming, a key subject in parallel and distributed computing courses. In this paper we provide a hands-on evaluation of verification and testing tools for concurrent programs, collecting a precise set of requirements, and describing to what extent they can or can not be used for this purpose. Our conclusion is that automatic grading of concurrent programming exercises remains an open challenge.

Modern I/O applications that run on HPC infrastructures are increasingly becoming read and metadata intensive. However, having multiple applications submitting large amounts of metadata operations can easily saturate the shared parallel file system's metadata resources, leading to overall performance degradation and I/O unfairness. We present PADLL, an application and file system agnostic storage middleware that enables QoS control of data and metadata workflows in HPC storage systems. It adopts ideas from Software-Defined Storage, building data plane stages that mediate and rate limit POSIX requests submitted to the shared file system, and a control plane that holistically coordinates how all I/O workflows are handled. We demonstrate its performance and feasibility under multiple QoS policies using synthetic benchmarks, real-world applications, and traces collected from a production file system. Results show that PADLL can enforce complex storage QoS policies over concurrent metadata-aggressive jobs, ensuring fairness and prioritization.

Performance of transactional systems is degraded by update hotspots as conflicts lead to waiting and wasted work. This is particularly challenging in emerging large-scale database systems, as latency increases the probability of conflicts, state-of-the-art lock-based mitigations are not available, and most alternatives provide only weak consistency and cannot enforce lower bound invariants. We address this challenge with Multi-Record Values (MRVs), a technique that can be layered on existing database systems and that uses randomization to split and access numeric values in multiple records such that the probability of conflict can be made arbitrarily small. The only coordination needed is the underlying transactional system, meaning it retains existing isolation guarantees. The proposal is tested on five different systems ranging from DBx1000 (scale-up) to MySQL GR and a cloud-native NewSQL system (scale-out). The experiments explore design and configuration trade-offs and, with the TPC-C and STAMP Vacation benchmarks, demonstrate improved throughput and reduced abort rates when compared to alternatives.

Approximate agreement has long been relegated to the sidelines compared to exact consensus, with its most notable application being clock synchronisation. Other proposed applications stemming from control theory target multi-agent consensus, namely for sensor stabilisation, coordination in robotics, and trust estimation. Several proposals for approximate agreement follow the Mean Subsequence Reduce approach, simply applying different functions at each phase. However, taking clock synchronisation as an example, applications do not fit neatly into the MSR model: Instead they require adapting the algorithms’ internals. Our contribution is two-fold. First, we identify additional configuration points, establishing a more general template of MSR approximate agreement algorithms. We then show how this allows us to implement not only generic algorithms but also those tailored for specific purposes (clock synchronisation). Second, we propose a toolkit for making approximate agreement practical, providing classical implementations as well as allow these to be configured for specific purposes. We validate the implementation with classical algorithms and clock synchronisation.

Cloud native database systems provide highly available and scalable services as part of cloud platforms by transparently replicating and partitioning data across automatically managed resources. Some systems, such as Google Spanner, are designed and implemented from scratch. Others, such as Amazon Aurora, derive from traditional database systems for better compatibility but disaggregate storage to cloud services. Unfortunately, because they follow an open-box approach and fork the original code base, they are difficult to implement and maintain. We address this problem with Loom, a replicated and partitioned database system built on top of PostgreSQL that delegates durable storage to a distributed log native to the cloud. Unlike previous disaggregation proposals, Loom is a closed-box approach that uses the original server through existing interfaces to simplify implementation and improve robustness and maintainability. Experimental evaluation achieves 6 × higher throughput and 5 × lower response time than standard replication and competes with the state of the art in cloud and HPC hardware.

EHealth has emerged as a new field at the intersection of medical informatics, public health and business, referring to healthcare services and information provided or enhanced through information technologies. Today, it is very focused on improving

There is an increasing demand for stateful edge computing for both complex Virtual Network Functions (VNFs) and application services in emerging 5G networks. Managing a mutable persistent state in the edge does however bring new architectural, performance, and dependability challenges. Not only it has to be integrated with existing cloud-based systems, but also cope with both operational and analytical workloads and be compatible with a variety of SQL and NoSQL database management systems. We address these challenges with AIDA-DB, a polyglot data management architecture for the edge and cloud continuum. It leverages recent development in distributed transaction processing for a reliable mutable state in operational workloads, with a flexible synchronization mechanism for efficient data collection in cloud-based analytical workloads.