Modern cyber-physical systems consolidate subsystems onto shared multi-core hardware to meet requirements in size, weight, power and cost. This practice introduces contention for shared hardware resources, which can break performance isolation and complicate timing predictability. While many timing verification techniques for partitioning individual resources such as CPU time, caches, interconnects and power budgets have been proposed over the past decades, real-time operating systems currently lack a scalable and universal mechanism for attaching multiple of these constraints to individual system workloads and incorporating this knowledge into thread scheduling decisions. Motivated by this observation, we present a scheduling model that uses scheduling constraints (SCs) as a first-class operating system primitive. We show that our model allows for the flexible specification of execution environments, resource partitioning plans, and even high-level application policies which get respected by the system scheduler. We implement SCs in the L4Re real-time framework and show that they impose minimal overhead, help to bound or mitigate contention for shared resources, and add temporal isolation to the strong logical isolation guarantees of L4Re’s microkernel-based design.

HPC networking is often characterized by kernel bypass, which is considered mandatory for large parallel and distributed applications. However, kernel bypass comes at a price because it breaks the traditional OS architecture, requiring applications to use special APIs and limiting the OS's control over existing network connections. We make the case that kernel bypass is not mandatory. Rather, high-performance networking relies on multiple performance-improving techniques, with kernel bypass even being detrimental to performance under specific conditions. CoRD removes kernel bypass from RDMA networks, primarily to enable efficient OS-level control over the RDMA dataplane. This control can be used to enhance security or resource allocation policies, and, as we demonstrate in one of the use cases, can improve end-to-end application performance by up to 10%. This architecture can enable Cloud-based distributed RDMA applications and facilitate deployment of coupled HPC applications.

Security mechanisms such as address spaces rely on the assumption that processor cores can be fully trusted. But the steady influx of side-channel vulnerabilities in processors is challenging this assumption. To minimize the impact of security vulnerabilities in processors, we need a system architecture that can tolerate potentially exploitable cores. In this paper, we propose the untrusted core isolation model to protect critical computation on trusted cores from untrusted and potentially buggy cores. We survey how current architectural building blocks such as MMUs fall short of this goal and derive requirements for untrusted core isolation. To demonstrate its feasibility, discuss both changes to commodity platforms and show how research works such as M3 fulfill the requirements. We evaluate the security benefits via a qualitative comparison of current architectures in both industry and academia and study its costs by a quantitative comparison of the most promising approaches on off-the-shelf and FPGA-based platforms.

Today, many applications are hosted by cloud providers. In order to isolate the workloads of different clients, cloud enterprises mostly rely on containers rather than standard processes, since the latter are able to exercise a lot of ambient authority. Containers counter this deficiency by sandboxing processes. To this end, they use dedicated security mechanisms such as seccomp-bpf. However, these mechanisms add complexity to the kernel and increase its attack surface, thus prompting new security challenges. Processes in microkernel-based systems do not have ambient authority. Thus, they do not require additional security mechanisms to build sandboxes. In this paper, we try to answer the question whether a microkernel-based OS architecture enables a leaner and more secure container infrastructure. Based on a CVE analysis, we show that the conceptual simplicity of containers on microkernels results in a better security posture than that typically found on monolithic systems. We furthermore demonstrate the practical feasibility of implementing containers on state-of-the-art microkernels by building MettEagle, a prototype container service running on L4Re. We found that applications running in containers on L4Re expose performance characteristics comparable to that of containers on Linux for both synthetic and real-world benchmarks. In some cases, the container implementation of L4Re even outperforms Linux, accelerating container startup latency and improving network performance.

Confidential computing based on Trusted Execution Environments (TEEs) allows software to run on remote servers without trusting the administrator. Remote attestation offers verifiable proof of the software stack and hardware elements comprising the TEE. However, setting up a secure channel to such a TEE requires a security guarantee that the channel actually terminates inside the TEE. TLS is an existing protocol for secure channel establishment, and in its most common use on the Web, it uses a key pair to assert the server identity encoded in a certificate. Various approaches have been proposed to integrate remote attestation into TLS. Unfortunately, they all have shortcomings. In this paper, we present a protocol that combines the existing certificate-based assurances of TLS with remote attestation-based assurances in away that they can be deployed independently and can fail independently. We design these two assurances to be additive without relying on each other, a property that has not been considered by existing approaches.

Asynchronous programming is a very helpful programming methodology for systems with many threads and long-running I/O. However, care must be taken if systems, e.g., kernels, work with objects that can be dynamically created, used, and destroyed by asynchronous tasks. The typical approach keeps an object alive until all tasks are done with it. However, this can lead to long delays until resources can be reclaimed and performs useless work on already destroyed objects. This paper proposes a new set of abstractions for object references that enables an immediate resource reclamation. We build upon the Rust programming language and use a combination of static analysis and runtime checks to guarantee that suspended tasks do not keep objects alive. We additionally leverage language features to ensure that destroyed objects cannot be accessed anymore, making it robust. We perform a case study with the M3 kernel, which has many asynchronous system calls and manages shared kernel objects. In the evaluation, we show that our approach incurs a modest performance overhead even for system-call heavy workloads.

Disaggregation is an ongoing trend to increase flexibility in datacenters. With interconnect technologies like CXL, pools of CPUs, accelerators, and memory can be connected via a datacenter fabric. Applications can then pick from those pools the resources necessary for their specific workload. However, this vision becomes less clear when we consider data movement. Workloads often require data to be streamed through chains of multiple devices, but typically, these data streams physically do not directly flow device-to-device, but are staged in memory by a CPU hosting device protocol logic. We show that augmenting devices with a disaggregation-native and device-independent data streaming facility can improve processing latencies by enabling data flows directly between arbitrary devices.

Containers are a light weight alternative to virtual machines, building on sandboxed processes whose permissions are restricted by additional security mechanisms such as seccomp-bpf. However, these mechanisms increase the kernel's attack surface, thus prompting new security challenges. In this paper, we ask the question of whether a system with processes properly restricted by design enables a container infrastructure with better security posture. For instance, microkernels with capability-based access control provide container-style isolation out of the box. On the basis of real-world CVEs, we argue that this conceptual simplicity actually results in a better security posture than that typically found on monolithic systems. We propose Oak, a container engine built on top of L4Re, a state-of-the-art microkernel-based operating system. For startup as well as for network microbenchmarks, containers running on L4Re exposed performance characteristics similar to that of containers on Linux. We thus conclude that building containers on microkernel is an approach worth pursuing further under both a performance and a security perspective.

Cloud environments, comprising both virtual and physical servers, are complex distributed systems that require clear and expressive configuration descriptions. Human-readable configuration formats like Kubernetes YAML are state of the art, but they lack the granularity needed for fine-grained control and advanced policy enforcement. To address these limitations, we propose an abstract system description approach that incorporates additional application properties, enabling more sophisticated policy decision-making rather than relying on resource constraints and port-based network restrictions. Our framework introduces two modes of policy enforcement: one allows system designers to automatically verify and manipulate system descriptions before translating them into concrete configurations, while the other enables communication partners to review the descriptions for assessing trustworthiness. We introduce a user-friendly description language paired with an extensible policy enforcement engine, providing stakeholders with the ability to define deployment scenarios intuitively and securely. We demonstrate the suitability of the approach for three different platforms, ranging from an embedded system to state-of-the-art container runtimes, namely Kubernetes and Docker Compose.

Our CPUs contain a compute instruction set, which regular applications use. But they also feature an intricate underworld of different CPU modes, combined with trap and exception handling to transition between these modes. These mechanisms are manifold and complex, yet the layering and functionality offered by the CPU modes is fixed. We have to take what CPU vendors provide, including potential security problems from unneeded modes. This paper explores the question, whether CPU modes could instead be defined entirely by software. We show how such a design would function and explore the advantages it enables. We believe that pushing all existing modes under a common design umbrella would enforce a cleaner structure and more control over exposed functionality. At the same time, the flexibility of software-defined modes enables interesting new use cases.

Whole system persistence (WSP) is a concept for retaining the computational state of a system even in case of a power failure. In the context of server systems, WSP could render it possible to quickly power on and off machines that only need to be used occasionally, thus saving energy. This paper takes on this idea and discusses multiple approaches for implementing WSP on such machines. Our evaluation shows that after starting a system, an NVM-based version of WSP can achieve tail latency improvements of up to 93% compared to booting a system and loading data from an SSD. At the same time, WSP is able to provide suspend and resume times in the order of tens of milliseconds.

Modern cyber-physical systems often require security, heterogeneity, and real-time operation from their hardware platform and operating system. However, highly predictable real-time operating systems such as FreeRTOS do not employ strong component isolation required for platform security. Microkernels implement such isolation using virtual memory and code running in the privileged CPU mode, complicating real-time analysis. In this work, we start with a different architectural approach: M3 is an existing hardware/software co-design for heterogeneous systems that features strong isolation between cores. However, the real-time properties of this platform have not been investigated. We first survey M3{\prime}s current state for real-time applicability and study both the communication latencies in comparison to other systems and M3's different approach to task priorities. Furthermore we improve M3's real-time applicability by adding a network-on-chip traffic regulation and enabling the enforcement of resource limits. With these additions, M3 enables local reasoning about application execution. We perform the evaluation with an FPGA-based hardware prototype and in simulation based on gem5.

State-of-the-art implementations of Trusted Execution Environments (TEEs) present system designers and users with several problems: First, it is not possible to choose a TEE implementation independently from the instruction set architecture. Second, the security-critical functionality of such TEEs is deeply integrated into the micro-architecture of complex processor cores, making programs running in such TEEs vulnerable to side-channel attacks. And third, the interface and execution model of certain types of TEEs make it hard to integrate these TEEs with the system software. To address these issues, we propose a modular TEE design. We apply this modular design to the M 3 hardware/software co-design platform and demonstrate how TEE support can be made a first-class feature at the system-architecture level.

High-performance networking is often characterized by kernel bypass which is considered mandatory in high-performance parallel and distributed applications. But kernel bypass comes at a price because it breaks the traditional OS architecture, requiring applications to use special APIs and limiting the OS control over existing network connections. We make the case, that kernel bypass is not mandatory. Rather, high-performance networking relies on multiple performance-improving techniques, with kernel bypass being the least effective. CoRD removes kernel bypass from RDMA networks, enabling efficient OS-level control over RDMA dataplane.

The continuously growing importance of today’s technology paradigms such as the Internet of Things (IoT) and the new 5G/6G standard open up unique features and opportunities for smart systems and communication devices. Famous examples are edge computing and network slicing. Generational technology upgrades provide unprecedented data rates and processing power. At the same time, these new platforms must address the growing security and privacy requirements of future smart systems. This poses two main challenges concerning the digital processing hardware. First, we need to provide integrated trustworthiness covering hardware, runtime, and the operating system. Whereas integrated means that the hardware must be the basis to support secure runtime and operating system needs under very strict latency constraints. Second, applications of smart systems cover a wide range of requirements where "one- chip-fits-all" cannot be the cost and energy effective way forward. Therefore, we need to be able to provide a scalable hardware solution to cover differing needs in terms of processing resource requirements.In this paper, we discuss our research on an integrated design of a secure and scalable hardware platform including a runtime and an operating system. The architecture is built out of composable and preferably simple components that are isolated by default. This allows for the integration of third-party hardware/software without compromising the trusted computing base. The platform approach improves system security and provides a viable basis for trustworthy communication devices.

We are approaching a world, where the CPU merely orchestrates a plethora of specialized devices such as accelerators, RDMA NICs, or non-volatile memory (NVM). Such devices operate by mapping their internal memory directly into an application’s address space for fast, low-latency access. With the latency of modern I/O devices low enough to make traditional system calls a performance bottleneck, kernel interaction has no place on the data path of microsecond-scale systems. However, kernel bypass prevents the OS from controlling and supervising access to the hardware. This paper tries to make a step back by bringing the OS to the critical path again, but with a reduced performance penalty. We pick up on previous ideas for reducing the cost of kernel interaction and propose the fastcall space , a new layer in the traditional OS architecture that hosts specialized and quickly accessible OS functions called fastcalls . Fastcalls can stay on the critical path of a microsecond-scale application because the invocation of fastcall-space functionality is up to 15 times faster than calling kernel functions from user space. We present and evaluate a prototype implementation of the fastcall framework and thereby show how much the overhead of calling into privileged mode can be reduced while using standard CPU features.

The M³ system (ASPLOS ’16) proposed a hardware/software co-design that simplifies integration between general-purpose cores and special-purpose accelerators, allowing users to easily utilize them in a unified manner. M³ is a tiled architecture, whose tiles (cores and accelerators) are partitioned between applications, such that each tile is dedicated to its own application. The M³x system (ATC ’19) extended M³ by trading off some isolation to enable coarse-grained multiplexing of tiles among multiple applications. With M³x, if source tile t₁ runs code of application p and sends a message m to destination tile t₂ while t₂ is currently not associated with p, then m is forwarded to the right place through a “slow path”, via some special OS tile. In this paper, we present M³v, which extends M³x by further trading off some isolation between applications to support “fast path” communication that does not require the said OS tile’s involvement. Thus, with M³v, a tile can be efficiently multiplexed between applications provided it is a general-purpose core. M³v achieves this goal by 1) adding a local multiplexer to each such core, and by 2) virtualizing the core’s hardware component responsible for cross-tile communications. We prototype M³v using RISC-V cores on an FPGA platform and show that it significantly outperforms M³x and may achieve competitive performance to Linux.

New 5G/6G mobile networks will allow to run distributed multi-tenancy workloads with high requirements on latency, throughput, and energy efficiency. This also demands a transformation of the underlying radio access networks (RANs) towards an architecture with open-source software, heterogeneous hardware, and interoperable interfaces as already described by the O-RAN ALLIANCE. In principle, virtualization of the RAN allows for defining additional interfaces which can be used for system verification and therefore can increase the trustworthiness of implementations. However, due to many security risks induced by the current O-RAN specification, it requires to rethink the overall security architecture from the ground up to establish trustworthy mobile networks in general.In this paper, we discuss hardware-enforced capabilities as a structuring principle for future O-RAN architectures. We present our approach of a hardware/operating system co-design to implement these principles. The evaluation results of architectural components demonstrate the technical feasibility of our approach and illustrate that security and low latency can be achieved simultaneously.