With the growing complexity of smartphone systems, effective tracing becomes vital for enhancing their stability and optimizing the user experience. Unfortunately, existing tracing tools are inefficient in smartphone scenarios. Their distributed designs (with either per-core or per-thread buffers) prioritize performance but lead to missing crucial clues with high probability. While these problems can be overlooked in previous scenarios (e.g., servers), they drastically limit the usefulness of tracing on smartphones. To enable efficient tracing on smartphones, we propose BTrace: a tracing tool that combines the performance benefits of per-core buffers with the capability of retaining longer continuous traces by partitioning a global buffer into multiple blocks, which are dynamically assigned to the most demanding cores. BTrace further gracefully handles unique requirements of modern smartphones, e.g., core oversubscription and resizing. BTrace has been deployed in production, recording an average of 2x continuous traces compared to the current best-performing tracer (Linux ftrace) and improving performance by 20%. Using BTrace, we successfully identified numerous bugs that require traces of long duration and are challenging to locate with existing tracers.

HPC networking is often characterized by kernel bypass, which is considered mandatory for large parallel and distributed applications. However, kernel bypass comes at a price because it breaks the traditional OS architecture, requiring applications to use special APIs and limiting the OS's control over existing network connections. We make the case that kernel bypass is not mandatory. Rather, high-performance networking relies on multiple performance-improving techniques, with kernel bypass even being detrimental to performance under specific conditions. CoRD removes kernel bypass from RDMA networks, primarily to enable efficient OS-level control over the RDMA dataplane. This control can be used to enhance security or resource allocation policies, and, as we demonstrate in one of the use cases, can improve end-to-end application performance by up to 10%. This architecture can enable Cloud-based distributed RDMA applications and facilitate deployment of coupled HPC applications.

Energy efficiency has become a key concern in modern computing. Major processor vendors now offer single-ISA heterogeneous processors that combine powerful and energy-efficient cores, such as Arm's big.LITTLE CPUs, Apple's M-series chips, and Intel P/E systems. However, today's OS schedulers, relying on simple cost-based thread allocation strategies, fail to fully exploit their potential. This paper presents HARP, a Linux-integrated resource-management framework for heterogeneous processors. HARP leverages application behavior through online monitoring or application descriptions and introduces a lightweight interface for two-way communication between applications and the resource manager. Through this interface, HARP learns application characteristics to guide allocation decisions, which are then relayed back to the applications so they can adapt accordingly. HARP supports various programming models, from OpenMP and Intel TBB to custom models with adaptivity features, significantly improving performance and energy efficiency, particularly in multi-application scenarios. On two representative heterogeneous systems, HARP reduces the average execution time by 12 % and the energy consumption by 28 % compared to existing methods. Overall, HARP marks a crucial step toward energy-efficient computing across diverse architectures.

Today, many applications are hosted by cloud providers. In order to isolate the workloads of different clients, cloud enterprises mostly rely on containers rather than standard processes, since the latter are able to exercise a lot of ambient authority. Containers counter this deficiency by sandboxing processes. To this end, they use dedicated security mechanisms such as seccomp-bpf. However, these mechanisms add complexity to the kernel and increase its attack surface, thus prompting new security challenges. Processes in microkernel-based systems do not have ambient authority. Thus, they do not require additional security mechanisms to build sandboxes. In this paper, we try to answer the question whether a microkernel-based OS architecture enables a leaner and more secure container infrastructure. Based on a CVE analysis, we show that the conceptual simplicity of containers on microkernels results in a better security posture than that typically found on monolithic systems. We furthermore demonstrate the practical feasibility of implementing containers on state-of-the-art microkernels by building MettEagle, a prototype container service running on L4Re. We found that applications running in containers on L4Re expose performance characteristics comparable to that of containers on Linux for both synthetic and real-world benchmarks. In some cases, the container implementation of L4Re even outperforms Linux, accelerating container startup latency and improving network performance.

Work stealing is a well-established technique in multi-core systems that aims to improve load balancing and task scheduling efficiency. Each processing unit maintains its own task queue, and when idle, it steals tasks from other units. Traditional work-stealing approaches face performance bottlenecks due to costly synchronization primitives and contention arising from concurrent access by both the queue owner and thieves. The state-of-the-art solution addresses these issues through coarse-grained synchronization; however, it restricts stealing in specific scenarios, thereby limiting parallelism. We introduce PadWS, a partial and asynchronous delegated work-stealing algorithm. PadWS employs a block-based design in which, under common cases, the queue owner and thieves work on separate blocks, reducing metadata contention. Delegation is partially enabled for the block in which the owner is located, allowing thieves to steal from it-an approach that deviates from the current block-based approach. Additionally, our delegation strategy is asynchronous, which removes the need for thieves to spin-wait after sending a request.

Energy efficiency has become a key concern in modern computing. Major processor vendors now offer heterogeneous architectures that combine powerful cores with energy-efficient ones, such as Intel P/E systems, Apple M1 chips, and Samsungs Exyno's CPUs. However, apart from simple cost-based thread allocation strategies, today's OS schedulers do not fully exploit these systems' potential for adaptive energy-efficient computing. This is, in part, due to missing application-level interfaces to pass information about task-level energy consumption and application-level elasticity. This paper presents E-Mapper, a novel resource management approach integrated into Linux for improved execution on heterogeneous processors. In E-Mapper, we base resource allocation decisions on high-level application descriptions that user can attach to programs or that the system can learn automatically at runtime. Our approach supports various programming models including OpenMP, Intel TBB, and TensorFlow. Crucially, E-Mapper leverages this information to extend beyond existing thread-to-core allocation strategies by actively managing application configurations through a novel uniform application-resource manager interface. By doing so, E-Mapper achieves substantial enhancements in both performance and energy efficiency, particularly in multi-application scenarios. On an Intel Raptor Lake and an Arm big.LITTLE system, E-Mapper reduces the application execution on average by 20 34 approach for sustainable and efficient computing across different processor architectures.

Containers are a light weight alternative to virtual machines, building on sandboxed processes whose permissions are restricted by additional security mechanisms such as seccomp-bpf. However, these mechanisms increase the kernel's attack surface, thus prompting new security challenges. In this paper, we ask the question of whether a system with processes properly restricted by design enables a container infrastructure with better security posture. For instance, microkernels with capability-based access control provide container-style isolation out of the box. On the basis of real-world CVEs, we argue that this conceptual simplicity actually results in a better security posture than that typically found on monolithic systems. We propose Oak, a container engine built on top of L4Re, a state-of-the-art microkernel-based operating system. For startup as well as for network microbenchmarks, containers running on L4Re exposed performance characteristics similar to that of containers on Linux. We thus conclude that building containers on microkernel is an approach worth pursuing further under both a performance and a security perspective.

Rising energy consumption is of growing concern for cloud data center providers. Modern processors try to counteract this problem through low-power idle states that save energy in phases with little demand for compute resources. Making proper use of this feature, however, requires knowledge about the properties of these states for the very processors used in a specific setup; most importantly, the energy consumed in each idle state and the latency for resuming normal operation. Unfortunately, hardware vendors usually do not provide this critical information. In this paper, we propose a scheme for automatically analyzing the idle states of modern Intel processors. Our open-source implementation uses an extensible Linux kernel module to measure the energy and latency implications of a system's processor without any manual intervention or external equipment. We demonstrate the practical applicability of our approach by analyzing two Intel processors from the Haswell and Skylake generation ---an Intel Core i7-4790 and an Intel Core i7-6700K, respectively. The results show that our implementation yields reliable, precise, and reproducible measurements for the energy and latency implications of each processor's various idle states.

Whole system persistence (WSP) is a concept for retaining the computational state of a system even in case of a power failure. In the context of server systems, WSP could render it possible to quickly power on and off machines that only need to be used occasionally, thus saving energy. This paper takes on this idea and discusses multiple approaches for implementing WSP on such machines. Our evaluation shows that after starting a system, an NVM-based version of WSP can achieve tail latency improvements of up to 93% compared to booting a system and loading data from an SSD. At the same time, WSP is able to provide suspend and resume times in the order of tens of milliseconds.

Work stealing is a widely-used scheduling technique for parallel processing on multicore. Each core owns a queue of tasks and avoids idling by stealing tasks from other queues. Prior work mostly focuses on balancing workload among cores, disregarding whether stealing may adversely impact the owner's performance or hinder synchronization optimizations. Real-world industrial runtimes for parallel processing heavily rely on work-stealing queues for scalability, and such queues can become bottlenecks to their performance. We present Block-based Work Stealing (BWoS), a novel and pragmatic design that splits per-core queues into multiple blocks. Thieves and owners rarely operate on the same blocks, greatly removing interferences and enabling aggressive optimizations on the owner's synchronization with thieves. Furthermore, BWoS enables a novel probabilistic stealing policy that guarantees thieves steal from longer queues with higher probability. In our evaluation, using BWoS improves performance by up to 1.25x in the Renaissance macrobenchmark when applied to Java G1GC, provides an average 1.26x speedup in JSON processing when applied to Go runtime, and improves maximum throughput of Hyper HTTP server by 1.12x when applied to Rust Tokio runtime. In microbenchmarks, it provides 8-11x better performance than state-of-the-art designs. We have formally verified and optimized BWoS on weak memory models with a model-checking-based framework.

High-performance networking is often characterized by kernel bypass which is considered mandatory in high-performance parallel and distributed applications. But kernel bypass comes at a price because it breaks the traditional OS architecture, requiring applications to use special APIs and limiting the OS control over existing network connections. We make the case, that kernel bypass is not mandatory. Rather, high-performance networking relies on multiple performance-improving techniques, with kernel bypass being the least effective. CoRD removes kernel bypass from RDMA networks, enabling efficient OS-level control over RDMA dataplane.

CPUs with weak memory-consistency models (WMMs), such as Arm and RISC-V, are rapidly increasing their market share. Porting legacy x86 applications to such CPUs requires introducing extra synchronization to prevent WMM-related concurrency bugs---a task often left to human experts. Given the rarity of such experts and the enormous size of legacy applications, we develop AtoMig, an effective, fully automated tool for porting large, real-world applications to WMM CPU architectures. AtoMig detects shared memory access patterns with novel static analysis strategies and performs program transformations to properly protect them from WMM effects. In the absence of sufficiently scalable verification methods, AtoMig shows practicality of focusing on code patterns more prone to WMM faults, trading off completeness for scalability. We validate the correctness of AtoMig's transformations on several small concurrent benchmarks via model checking. We demonstrate the scalability and performance of our approach by applying AtoMig to popular real-world large code bases with up to millions of lines of code, viz., MariaDB, Postgres, SQlite, LevelDB, and Memcached. As part of this work, we also found a WMM bug in MariaDB, which AtoMig fixes automatically.

Future industrial networks will consist of a complex mixture of new and legacy components, while new use cases and applications envisioned by Industry 4.0 will demand increased flexibility and dynamics from these networks. Industrial security gateways will become an important building block to tackle new security requirements demanded by these changes. Their introduction will further increase the already high complexity of these networks, demanding more efforts in properly and securely configuring them. Yet, past research showed, that most operators of industrial networks are already today unable to configure industrial networks in a secure fashion. Therefore, we propose a scheme that allows factory operators to configure security gateways in an easy and practical way that is also understandable for staff not trained in the security domain. We employ hardware security tokens that allow to reduce every day configuration to one physical interaction. Our results show the practical feasibility of our proposed scheme and that it does not reduce the security level of industrial security gateways in any way.

Contemporary HPC systems use batch scheduling of compute jobs running on exclusively assigned hardware resources. During communication, polling for progress is the state of the art as it promises minimal latency. Previous work on oversubscription and event-based communication, i.e. vacating the CPU while waiting for communication to finish, shows that these techniques can improve the overall system utilisation and reduce the energy consumption. Despite these findings, neither of the two techniques is commonly used in HPC systems today. We believe that the current lack of detailed studies of the low-level effects of event-based communication, a key enabler of efficient oversubscription for classical MPI-based applications, is a major obstacle to a wider adoption. We demonstrate that the sched_yield system call, which is often used for oversubscription scenarios, is not best suited for this purpose on modern Linux systems. Furthermore, we incorporate event-based communication into Open MPI and evaluate the effects on latency and energy consumption using an MPI micro-benchmark. Our results indicate that event-base communication incurs significant latency overhead but also saves energy. Both effects grow with the imbalance of the application using MPI.

Trends like Industry 4.0 will pose new challenges for future industrial networks. Greater interconnectedness, higher data volumes as well as new requirements for speeds as well as security will make new approaches necessary. Performanceoptimized networking techniques will be demanded to implement new use cases, like network separation and isolation, in a secure fashion. A new and highly efficient protocol, that will be vital for that purpose, is MACsec. It is a Layer 2 encryption protocol that was previously extended specifically for industrial environments. Yet, it lacks the ability to bridge local networks. Therefore, in this work, we propose a secure and efficient Layer 3 tunneling scheme for MACsec. We design and implement two approaches, that are equally secure and considerably outperform comparable state-of-the-art techniques.

Concurrent bounded queues have been widely used for exchanging data and profiling in operating systems, databases, and multithreaded applications. The performance of state-of-the-art queues is limited by the interference between multiple enqueues (enq-enq), multiple dequeues (deq-deq), or enqueues and dequeues (enq-deq), negatively affecting their latency and scalability. Although some existing designs employ optimizations to reduce deq-deq and enq-enq interference, they often neglect the enq-deq case. In fact, such partial optimizations may inadvertently increase interference elsewhere and result in performance degradation. We present Block-based Bounded Queue (BBQ), a novel ringbuffer design that splits the entire buffer into multiple blocks. This eliminates enq-deq interference on concurrency control variables when producers and consumers operate on different blocks. Furthermore, the block-based design is amenable to existing optimizations, e.g., using the more scalable fetch-and-add instruction. Our evaluation shows that BBQ outperforms several industrial ringbuffers. For example, in single-producer/single-consumer micro-benchmarks, BBQ yields 11.3x to 42.4x higher throughput than the ringbuffers from Linux kernel, DPDK, Boost, and Folly libraries. In real-world scenarios, BBQ achieves up to 1.5x, 50.5x, and 11.1x performance improvements in benchmarks of DPDK, Linux io_uring, and Disruptor, respectively. We verified and optimized BBQ on weak memory models with a model-checkingbased framework.

We are approaching a world, where the CPU merely orchestrates a plethora of specialized devices such as accelerators, RDMA NICs, or non-volatile memory (NVM). Such devices operate by mapping their internal memory directly into an application’s address space for fast, low-latency access. With the latency of modern I/O devices low enough to make traditional system calls a performance bottleneck, kernel interaction has no place on the data path of microsecond-scale systems. However, kernel bypass prevents the OS from controlling and supervising access to the hardware. This paper tries to make a step back by bringing the OS to the critical path again, but with a reduced performance penalty. We pick up on previous ideas for reducing the cost of kernel interaction and propose the fastcall space , a new layer in the traditional OS architecture that hosts specialized and quickly accessible OS functions called fastcalls . Fastcalls can stay on the critical path of a microsecond-scale application because the invocation of fastcall-space functionality is up to 15 times faster than calling kernel functions from user space. We present and evaluate a prototype implementation of the fastcall framework and thereby show how much the overhead of calling into privileged mode can be reduced while using standard CPU features.

Disaggregated heterogeneous data centers promise higher efficiency, lower total costs of ownership, and more flexibility for data-center operators. However, current software stacks can levy a high tax on application performance. Applications and OSes are designed for systems where local PCIe-connected devices are centrally managed by CPUs, but this centralization introduces unnecessary messages through the shared data-center network in a disaggregated system. We present FractOS, a distributed OS that is designed to minimize the network overheads of disaggregation in heterogeneous data centers. FractOS elevates devices to be first-class citizens, enabling direct peer-to-peer data transfers and task invocations among them, without centralized application and OS control. FractOS achieves this through: (1) new abstractions to express distributed applications across services and disaggregated devices, (2) new mechanisms that enable devices to securely interact with each other and other data-center services, (3) a distributed and isolated OS layer that implements these abstractions and mechanisms, and can run on host CPUs and SmartNICs. Our prototype shows that FractOS accelerates real-world heterogeneous applications by 47%, while reducing their network traffic by 3×.

Industry 4.0 will revolutionize industrial automation. Yet, future smart factories will not be created from scratch. They will rather evolve from existing legacy installations. Consequently, also industrial networks will evolve and the result will be a mixture of new and legacy components. This will make new security mechanisms necessary, that are specifically designed for this industrial use case. This work proposes modifications for MACsec [1], a new security protocol for protecting communication traffic. These modifications enable MACsec to work within future industrial settings, circumventing drawbacks introduced by legacy networking technologies. Furthermore, we managed to significantly increase the performance of MACsec.

Major data centre providers are introducing RDMA-based networks for their tenants, as well as for operating the underlying infrastructure. In comparison to traditional socket-based network stacks, RDMA-based networks offer higher throughput, lower latency and reduced CPU overhead. However, transparent checkpoint and migration operations become much more difficult. The key reason is that the OS is removed from the critical path of communication. As a result, some of the communication state itself resides in the NIC hardware and is no more under the direct control of the OS. This control includes especially the support for virtualisation of communication which is needed for live migration of communication partners. In this paper, we propose the basic principles required to implement a migration-capable RDMA-based network. We recommend some changes at the software level and small changes at the hardware level. As a proof of concept, we integrate the proposed changes into SoftRoCE, an open-source kernel-level implementation of the RoCE protocol. We claim that these changes introduce no runtime overhead when migration does not happen. Finally, we develop a proof-of-concept implementation for migrating containerised applications that use RDMA-based networks.