As computing systems approach the limits of traditional silicon technology, the diminishing returns in performance per watt present a significant barrier to sustaining growth in HPC. From a large-scale scientific supercomputing facility point of view, we propose a multifaceted strategy toward specialized hardware and architectures that are optimized for energy efficiency in specific applications. We also emphasize the need for integrating energy-aware practices across all levels of HPC, from system design and software development to operational policies. We discuss strategic opportunities such as the adoption of application-specific accelerators, the development of energy-efficient algorithms, and the implementation of data-driven operational analytics. Our goal is to develop a comprehensive roadmap ensuring that future leadership systems at OLCF can meet scientific demands while operating within stringent energy budgets, thereby supporting sustainable computing growth.

Cloud platforms offer a variety of benefits that are very appealing for a large scale HPC facility with a diverse and dynamic user base and workload set. At the same time, there is cause for concern about transitioning to the cloud. Incorporating cloud resources into existing HPC facilities or even fully transitioning to a cloud deployment poses significant challenges at the technical, organizational, and economic levels. Regardless, based on current trends it is highly likely that cloud platforms will become an integral component of many HPC centers in some form. To gain a better understanding of both the limitations and capabilities of current cloud infrastructures we evaluated the public offerings of the three leading cloud platforms (Amazon Web Services, Microsoft Azure, and Google Cloud Platform) using a selection of representative application workloads from our facility. Our findings show that while current HPC offerings are still nascent, significant progress is being made to address the present shortcomings. At the same time, significant challenges and questions remain about whether HPC cloud offerings will be able to deliver the full range of expected benefits.

A federated instrument-to-edge-to-center architecture is needed to autonomously collect, transfer, store, process, curate, and archive scientific data and reduce human-in-the-loop needs with (a) common interfaces to leverage community and custom software, (b) pluggability to permit adaptable solutions, reuse, and digital twins, and (c) an open standard to enable adoption by science facilities world-wide. The Selfdriven Experiments for Science/Interconnected Science Ecosystem (INTERSECT) Open Architecture enables science breakthroughs using intelligent networked systems, instruments and facilities with autonomous experiments, “self-driving” laboratories, smart manufacturing and artificial intelligence (AI) driven design, discovery and evaluation. It creates an open federated architecture for the laboratory of the future using a novel approach, consisting of (1) science use case design patterns, (2) a system of systems architecture, and (3) a microservice architecture.

Oak Ridge National Laboratory (ORNL)'s Self-driven Experiments for Science / Interconnected Science Ecosystem (INTERSECT) architecture project, titled "An Open Federated Architecture for the Laboratory of the Future", creates an open federated hardware/software architecture for the laboratory of the future using a novel system of systems (SoS) and microservice architecture approach, connecting scientific instruments, robot-controlled laboratories and edge/center computing/data resources to enable autonomous experiments, "self-driving" laboratories, smart manufacturing, and artificial intelligence (AI)-driven design, discovery and evaluation.The architecture project is divided into three focus areas: design patterns; SoS architecture; and microservices architecture. The design patterns area focuses on describing science use cases as design patterns that identify and abstract the involved hardware/software components and their interactions in terms of control, work and data flow. The SoS architecture area focuses on an open architecture specification for the federated ecosystem that clarifies terms, architectural elements, the interactions between them and compliance. The microservices architecture describes blueprints for loosely coupled microservices, standardized interfaces, and multi-programming language support.This document is the SoS Architecture specification only, and captures the system of systems architecture design for the INTERSECT Initiative and its components. It is intended to provide a deep analysis and specification of how the INTERSECT platform will be designed, and to link the scientific needs identified across disciplines with the technical needs involved in the support, development, and evolution of a science ecosystem.This working document reflects current discussions and design activity among the authors. The authors have worked to eliminate significant inconsistencies in publicly available versions. Comments from readers are welcome as we continue to evolve this document and the INTERSECT SoS design it describes.

With the rising popularity of file-sharing services such as Google Drive and Dropbox in the workflows of individuals and corporations alike, the protection of client-outsourced data from unauthorized access or tampering remains a major security concern. Existing cryptographic solutions to this problem typically require server-side support, involve non-trivial key management on the part of users, and suffer from severe re-encryption penalties upon access revocations. This combination of performance overheads and management burdens makes this class of solutions undesirable in situations where performant, platform-agnostic, dynamic sharing of user content is required. We present NEXUS, a stackable filesystem that leverages trusted hardware to provide confidentiality and integrity for user files stored on untrusted platforms. NEXUS is explicitly designed to balance security, portability, and performance: it supports dynamic sharing of protected volumes on any platform exposing a file access API without requiring server-side support, enables the use of fine-grained access control policies to allow for selective sharing, and avoids the key revocation and file re-encryption overheads associated with other cryptographic approaches to access control. This combination of features is made possible by the use of a client-side Intel SGX enclave that is used to protect and share NEXUS volumes, ensuring that cryptographic keys never leave enclave memory and obviating the need to reencrypt files upon revocation of access rights. We implemented a NEXUS prototype that runs on top of the AFS filesystem and show that it incurs ×2 overhead for a variety of common file and database operations.

The increasing popularity and ubiquity of various large graph datasets has caused renewed interest for graph partitioning. Existing graph partitioners either scale poorly against large graphs or disregard the impact of the underlying hardware topology. A few solutions have shown that the nonuniform network communication costs may affect the performance greatly. However, none of them considers the impact of resource contention on the memory subsystems (e.g., LLC and Memory Controller) of modern multicore clusters. They all neglect the fact that the bandwidth of modern high-speed networks (e.g., Infiniband) has become comparable to that of the memory subsystems. In this paper, we provide an in-depth analysis, both theoretically and experimentally, on the contention issue for distributed workloads. We found that the slowdown caused by the contention can be as high as 11x. We then design an architecture-aware graph partitioner, Argo, to allow the full use of all cores of multicore machines without suffering from either the contention or the communication heterogeneity issue. Our experimental study showed (1) the effectiveness of Argo, achieving up to 12x speedups on three classic workloads: Breadth First Search, Single Source Shortest Path, and PageRank; and (2) the scalability of Argo in terms of both graph size and the number of partitions on two billion-edge real-world graphs.

Because of the increasing complexity of the applications running in Kitten, a lightweight HPC OS targeted for compute nodes of massively-parallel, distributed-memory supercomputers, and the complex hardware that Kitten is running on, bugs are becoming more difficult to find. As a result, the need for Kitten to support user-level application debugging becomes more critical. Unfortunately, Kitten currently has no support for user-level application debugging. To solve this problem, we implemented a GDB stub as a kernel module for Kitten without changing the internal architecture of Kitten and Palacios, a high performance embeddable VMM for Kitten, yet with minimal modification to Kitten and Palacios codebases. The GDB stub has only 590 lines of changes to Kitten and 238 lines of changes to Palacios in total. Along with the GDB stub and Palacios, we are now able to debug a userlevel application running in Kitten from a GDB client running in a Palacios Linux Guest. This paper presents our design and implementation of the GDB stub in Kitten.

This paper describes the design of a system to enable research, development, and testing of new software stacks and hardware features for future high-end computing systems. Motivating uses include both small-scale research and development on simulated individual nodes of proposed high-performance computing systems, and large scaling studies that emulate a sizeable fraction of a future supercomputing system. The proposed architecture combines system virtualization, architectural simulation, time dilation, and slack simulation to provide scalable emulation of hypothetical systems. Virtualization-based full-system measurement and monitoring tools are also included to aid in using the proposed system for co-design of high-performance computing system software and architectural features for future systems. Finally, this paper provides a description of the implementation strategy and status of the system.

A commodity I/O device has no support for virtualization. A VMM can assign such a device to a single guest with direct, fast, but insecure access by the guest's native device driver. Alternatively, the VMMcan build virtual devices on top of the physical device, allowing it to be multiplexed across VMs, but with lower performance. We propose a technique that provides an intermediate option. In virtual passthrough I/O (VPIO), the guest interacts directly with the physical device most of the time, achieving high performance, as in passthrough I/O. Additionally, the guest/device interactions drive a model that in turn identifies (1) when the physical device can be handed off to another VM, and (2) if the guest programs the device to behave illegitimately. In this paper, we describe the VPIO model, and present preliminary results in using it to support two commodity network cards within the Palacios VMM we are building. We believe that an appropriate model for an I/O device could be produced by the hardware vendor as part of the design, implementation, and testing process.

A commodity I/O device has no support for virtualization. A VMM can assign such a device to a single guest with direct, fast, but insecure access by the guest's native device driver. Alternatively, the VMM can build virtual devices on top of the physical device, allowing it to be multiplexed across VMs, but with lower performance. We propose a technique that provides an intermediate option. In virtual passthrough I/O (VPIO), the guest interacts directly with the physical device most of the time, achieving high performance, as in passthrough I/O. Additionally, the guest/device interactions drive a model that in turn identifies (1) when the physical device can be handed off to another VM, and (2) if the guest programs the device to behave illegitimately. In this paper, we describe the VPIO model, and present preliminary results in using it to support a commodity network card within the Palacios VMM we are building. We believe that an appropriate model for an I/O device could be produced by the hardware vendor as part of the design, implementation, and testing process.

In most existing computing environments, resources such as CPU time, memory, disk space, and network bandwidth are not used to capacity. In future environments, such as computational grids, resources may exist solely for sale. There is no clean way for a resource provider to sell either kind of resource because the current abstractions an owner can provide (and a buyer can request) are (1) too high level, (2) numerous, and (3) complex. Virtuoso is a prototype system that creates a marketplace in which resources can be straightforwardly sold by resource providers to resource buyers. The resource exchange is in the form of a low level virtual machine and virtual network that presents the abstraction of a new raw physical machine on the buyer's network. This report describes the interface and implementation of the Virtuoso system. It is also a user manual for those who wish to try Virtuoso. Effort sponsored by the National Science Foundation under Grants ANI-0093221, ACI- 0112891, ANI-0301108, EIA-0130869, and EIA-0224449. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation (NSF).

Packet headers and trailers exhibit considerable coding redundancy from both a theoretical and a practical standpoint. Diffusion exploits this redundancy to create an additional communication channel between hosts as a by-product of normal packet transfers. This channel is zero cost: the number and size of packets transferred do not change. Information is piggybacked on existing packets by overwriting unused fields in their headers and trailers. Dissemination of dynamic resource information is a natural use of such a channel. This paper describes the interface, implementation, and performance of Diffusion on the Linux operating system. In addition, it describes two tools, SpyTalk and LoadBanner, which have been built on top of this implementation. It is also a user manual for those who wish to try Diffusion.