The IO500 benchmark has become the community standard for evaluating HPC storage system performance, yet the detailed data contained in its submission packages remains largely unexplored beyond aggregate leaderboard rankings. We present a statistical characterization of 61 IO500 submissions from four competition lists (ISC21 through SC22), examining score distributions, inter-phase correlations, and insights derived from detailed log files that accompany each submission. Our analysis reveals that IO500 scores span four orders of magnitude. Spearman correlation analysis shows strong within-domain clustering for both bandwidth (rs = 0.78 to 0.96) and metadata (rs = 0.89 to 0.98) phases, with the composite sub-scores exhibiting rs = 0.92 at per-node level (Pearson r = 0.53). Log-level analysis uncovers file-system-specific patterns in IOR close-time overhead, straggler behavior during the stonewall wear-down phase, and parallel-find load imbalance that are invisible in aggregate scores. These findings demonstrate that IO500 submission packages constitute a valuable research resource for understanding storage system behavior. The full submission dataset is publicly available at https://github.com/IO500/submission-data, and analysis scripts at https://gitlab-ce.gwdg.de/hpc-team/io500-analysis.

When we think of how we use smartphones, e-commerce, collaboration platforms, LLMs, etc., most of our interactions with computers are interactive and often urgent. Similar trends of interactivity and urgency are coming to HPC, with applications from simulations to data analysis and machine learning requiring more parallel computational capability and more interactivity. This chapter overviews the progress made so far along with some vectors of what the path forward will bring for greater integration of interactive and urgent HPC policies, techniques, and technologies into our HPC ecosystems.

Recent developments indicate a shift toward web services that employ ever-larger AI models, e.g., large language models (LLMs), requiring powerful and expensive hardware for inference. High-performance computing (HPC) systems are commonly equipped with such hardware for the purpose of large-scale computation tasks. However, HPC infrastructure is inherently unsuitable for hosting real-time web services due to network, security, and scheduling constraints. While various efforts exist to integrate external scheduling solutions, these often require compromises in terms of security or usability for existing HPC users. In this paper, we present SAIA, a Slurm-native platform consisting of a scheduler and a proxy. The scheduler interacts with Slurm to ensure the availability and scalability of services, while the proxy provides external access, which is secured via confined SSH commands. We have demonstrated SAIA’s applicability by deploying a large-scale LLM web service that has served over 100 000 users.

Heterogeneous HPC workflow scheduling under multiple hard constraints poses a challenging combinatorial optimization problem. Classical exact solvers guarantee optimality but face scalability limits, motivating interest in quantum-inspired Quadratic Unconstrained Binary Optimization (QUBO) as an alternative optimization paradigm. This work presents a systematic empirical evaluation of QUBO-based scheduling methods against classical baselines including MILP, CP-SAT, GA, and HEFT. We evaluate three QUBO variants, single-run simulated annealing, multi-attempt annealing, and a layered QAOA-inspired schedule, with hybrid enhancement strategies on validation workflows (3-4 tasks) and synthetic scaling instances (5-20 tasks). All solvers are assessed through a unified pipeline tracking feasibility, makespan, and resource utilization under progressive constraint activation and controlled penalty sweeps. All approaches recover the expected optimal makespan on validation instances, confirming formulation correctness. However, feasibility degradation emerges for specific QUBO variants as constraint interactions intensify, particularly when communication costs are introduced. Penalty analysis reveals a sharp feasibility threshold for QUBO-SA, where insufficient penalties consistently fail and moderate-to-strong penalties restore feasibility. Scaling experiments show that classical solvers remain robust across all tested sizes, while QUBO-SA loses feasibility beyond 15 tasks and the QAOA-inspired variant beyond 10 tasks. The study provides a clear empirical characterization of the reliability boundaries of quantum-inspired QUBO formulations for HPC scheduling and identifies regimes where classical approaches remain preferable under current solver capabilities.

Predicting application I/O time in large HPC clusters is a difficult task due to the shared nature of the resources involved and intermingled layers of the I/O stack. However, a more accurate prediction of application I/O time would improve resource scheduling and make it easier to identify scaling and portability bottlenecks. Currently, this type of prediction, which involves high-dimensional parameter spaces, is often performed by using machine learning, introducing another problem: data availability. Several related works require a large amount of data or access to the specific cluster's job scripts to perform the prediction, which can be a significant challenge for small and medium clusters or new systems without prior performance data. In this work, we use a transfer learning workflow and I/O modeling to create a targeted I/O time prediction tailored for a specific application. Our work demonstrates that we can utilize similar to 1% of the data required to create the base model prediction to achieve up to 87.25% of the predictions that fall within the acceptable range of I/O time variability.

Modern high-performance computing (HPC) Input/Output (I/O) systems consist of stacked hard- and software layers that provide interfaces for data access. Depending on application needs, developers usually choose higher layers with richer semantics for the ease of use or lower layers for performance. Each I/O interface on a given stack consists of a set of operations and their syntactic definition, as well as a set of semantic properties. To properly function, high-level libraries such as Hierarchical Data Format version 5 (HDF5) need to map their semantics to lower-level Application Programming Interface (API) such as Portable Operating System Interface (POSIX). Lower-level storage backends provide different I/O semantics than the layers in the stack above while sometimes implementing the same interface. However, most I/O interfaces do not transport semantic information through their APIs. Ideally, no semantics of an I/O operation should be lost while passing through the I/O stack, allowing lower layers to optimize performance. Unfortunately, there is a lack of general definition and unified taxonomy of I/O semantics. Similarly, system-level APIs offer little support for passing semantics to underlying layers. Thus, passing semantic information between layers is currently not feasible. In this article, we systematically compare I/O interfaces by examining their semantics across the HPC I/O stack. Our primary goal is to provide a taxonomy and comparative analysis, not to propose a new I/O interface or implementation. We propose a general definition of I/O semantics and present a unified classification of I/O semantics based on the categories of concurrent access, persistency, consistency, spatiality, temporality, and mutability. This allows us to compare I/O interfaces in terms of their I/O semantics. We show that semantic information is lost while traveling through the storage stack, which often prevents the underlying storage backends from making the proper performance and consistency decisions. In other words, each layer acts like a semantic filter for the lower layers. We discuss how higher-level abstractions could propagate their semantics and assumptions down through the lower-levels of the I/O stack. As a possible mitigation, we discuss the conceptual design of semantics-aware interfaces, to illustrate how such interfaces might address semantic loss—though we do not propose a concrete new implementation.

The convergence of IoT, edge, cloud, and HPC technologies creates a heterogeneous compute continuum requiring sophisticated workload management. Current tools like SLURM, Kubernetes, and Snakemake lack automated optimization for cross-platform resource allocation, forcing users to manually map workloads across diverse infrastructures. We present a comprehensive framework integrating heterogeneous system and workload modeling integration with Snakemake followed by different tools and techniques like Mixed Integer Linear Programming (MILP) for multi-objective optimization to automate task mapping and scheduling across the compute continuum. Our approach extends Snakemake scheduler with formal mathematical models that optimize resource utilization and makespan. Experimental evaluation demonstrates that MILP-based solution achieves optimal scheduling for small-scale workflows (5x5 tasks) in 0.02 seconds, while heuristic methods provide 99.9% faster solutions for large-scale scenarios (5000x5000 tasks) with only 5-10% deviation from optimal makespan. For parallel workflows, the optimization achieves up to 16.7% makespan reduction compared to sequential scheduling approaches.

With the rise of LLMs, GPU acceleration has become essential for both training and serving AI models. This requires HPC systems to be highly flexible with assigning multi-GPU nodes while also maintaining high security standards. Existing approaches involve utilizing nodes with batch and service schedulers, e.g., Slurm and Kubernetes, by dynamically moving nodes between the schedulers either through negotiation between the systems or via an external system. However, such a multi-use approach also increases the attack surface as more scheduling components operate with root permission. Moreover, it becomes increasingly difficult to recover from a security incident as attackers might have infected parts of either scheduling system. In this work, we present Ephemeral Kubernetes as a way to dynamically deploy and remove Kubernetes clusters in Warewulf managed environments such that nodes can be booted to be either part of a Slurm or Kubernetes cluster while being wiped at shutdown.

Large language models (LLMs) are increasingly explored for their reasoning capabilities, yet their ability to perform structured, constraint-based optimization from natural language remains insufficiently understood. This study evaluates twenty-one publicly available LLMs on a representative heterogeneous high-performance computing (HPC) workload mapping and scheduling problem. Each model received the same textual description of system nodes, task requirements, and scheduling constraints, and was required to assign tasks to nodes, compute the total makespan, and explain its reasoning. A manually derived analytical optimum of nine hours and twenty seconds served as the ground truth reference. Three models exactly reproduced the analytical optimum while satisfying all constraints, twelve achieved near-optimal results within two minutes of the reference, and six produced suboptimal schedules with arithmetic or dependency errors. All models generated feasible task-to-node mappings, though only about half maintained strict constraint adherence. Nineteen models produced partially executable verification code, and eighteen provided coherent step-by-step reasoning, demonstrating strong interpretability even when logical errors occurred. Overall, the results define the current capability boundary of LLM reasoning in combinatorial optimization: leading models can reconstruct optimal schedules directly from natural language, but most still struggle with precise timing, data transfer arithmetic, and dependency enforcement. These findings highlight the potential of LLMs as explainable co-pilots for optimization and decision-support tasks rather than autonomous solvers.

AdamZ is an advanced variant of the Adam optimiser, developed to enhance convergence efficiency in neural network training. This optimiser dynamically adjusts the learning rate by incorporating mechanisms to address overshooting and stagnation, which are common challenges in optimisation. Specifically, AdamZ reduces the learning rate when overshooting is detected and increases it during periods of stagnation, utilising hyperparameters such as overshoot and stagnation factors, thresholds, and patience levels to guide these adjustments. While AdamZ may lead to slightly longer training times compared to some other optimisers, it consistently excels in minimising the loss function, making it particularly advantageous for applications where precision is critical. Benchmarking results demonstrate the effectiveness of AdamZ in maintaining optimal learning rates, leading to improved model performance across diverse tasks.

This paper presents a systematic review of mapping and scheduling strategies within the High-Performance Computing (HPC) compute continuum, with a particular emphasis on heterogeneous systems. It introduces a prototype workflow to establish foundational concepts in workload characterization and resource allocation. Building on this, a thorough analysis of 66 selected research papers - spanning the period from 2017 to 2024 - is conducted, evaluating contemporary tools and techniques used for workload mapping and scheduling. The review highlights that conventional Job Shop scheduling formulations often lack the expressiveness required to model the complexity of modern HPC data centers effectively. It also reaffirms the classification of HPC scheduling problems as NP-hard, due to their combinatorial nature and the diversity of system and workload constraints. The analysis reveals a prevailing reliance on heuristic and meta-heuristic strategies, including nature-inspired, evolutionary, sorting, and search algorithms. To bridge the observed gaps, the study advocates for hybrid optimization approaches that strategically integrate heuristics, meta-heuristics, machine learning, and emerging quantum computing techniques. Such integration, when tailored to specific problem domains, holds promise for significantly improving the scalability, efficiency, and adaptability of workload optimization in heterogeneous HPC environments.

Kubernetes has emerged as the industry standard for container orchestration in cloud environments, with its scheduler dynamically placing container instances across cluster nodes based on predefined rules and algorithms. Various efforts have been made to extend and improve upon the Kubernetes scheduler. However, as the majority of Kubernetes clusters operate on homogeneous hardware, most scheduling algorithms are also only developed for homogeneous systems. Heterogeneous infrastructures, which include IoT devices or specialized hardware, have become more widespread and require specialized tuning to optimize workload assignment, for which researchers and developers working on scheduling systems require access to heterogeneous hardware for development and testing; such data may not be available. While simulations such as CloudSim or K8sSim can provide insights, the level of detail they can offer to validate new schedulers is limited, as they are only simulations. To address this, we introduce Q8S, a tool for emulating heterogeneous Kubernetes clusters including x86_64 and ARM64 architectures on OpenStack using QEMU. Emulations created through Q8S provide a higher level of detail than simulations and can be used to train machine learning scheduling algorithms. By providing an environment capable of executing real workloads, Q8S enables researchers and developers to test and refine their scheduling algorithms, ultimately leading to more efficient and effective heterogeneous cluster management. We release our implementation of Q8S as open source.

Providing a high-quality performance prediction has the potential to enhance various aspects of a cluster, such as devising scheduling and provisioning policies, guiding procurement decisions, suggesting candidate applications for tuning, and identifying probable scaling and porting challenges. Creating such a prediction for the I/O metrics is still challenging, however, due to the intricate interplay of multiple cluster components, making this an ideal case for machine learning. Nevertheless, achieving the required accuracy level with machine learning calls for a substantial amount of high-quality data, which is often a difficult challenge for most HPC clusters. In this work we explore the use of transfer learning to predict the applications’ I/O bandwidth based on a public dataset. As a result, our experiment can provide an I/O bandwidth prediction for a different cluster comparable to the current state-of-the-art result while employing 100 times less data than needed to construct the base model. Furthermore, we evaluate potential future improvements of the proposed workflow.

Cloud computing clusters, especially those handling cloud workloads, require efficient job scheduling to optimize resource utilization and minimize completion time. Traditional approaches often fall short in dynamic cloud environments. We propose “HOSHMAND” (High-performance Open sourced AI-based Scheduling Handler for MAnaging Node Distribution), an AI-driven framework using a custom-tailored Recurrent Neural Network (RNN) to rapidly predict the most suitable nodes for cloud workload execution. A key feature of HOSHMAND is its accelerated scheduling capability, which significantly reduces the time required for job allocation compared to traditional methods. This is particularly crucial for cloud environments with fluctuating workloads and diverse computational requirements. A distinct capability of HOSHMAND is its proficiency in managing heterogeneous resources, ensuring optimal allocation regardless of varying computational capabilities or resource types. This adaptability is crucial for contemporary cloud computing en-vironments, which often comprise a diverse array of hardware configurations, to maintain high efficiency and resource utilization. Moreover, HOSHMAND mitigates the overhead associated with repetitive scheduling computations in similar scenarios by leveraging its historical knowledge. Upon recognizing a con-figuration of jobs analogous to previously encountered situations, it promptly enacts the most effective scheduling strategy without redundant recalculations. This predictive capability not only conserves computational resources but also accelerates job execution. Our approach, tested on cloud-based datasets, demonstrates remarkable improvements in scheduling speed and efficiency, validated by reduced time-to-schedule and enhanced overall system throughput. Through its innovative handling of heterogeneous resources and intelligent avoidance of unnecessary scheduling computations, HOSHMAND sets a new benchmark for AI -driven job scheduling in cloud computing environments.

As a broader set of applications from simulations to data analysis and machine learning require more parallel computational capability, the demand for interactive and urgent high performance computing (HPC) continues to increase. This paper overviews the progress made so far and elucidates the challenges and opportunities for greater integration of interactive and urgent HPC policies, techniques, and technologies into HPC ecosystems.

We present our workflow and research data management (RDM) within the HeartAndBrain research project of the Department of Neurology at the University Medical Center Göttingen.Here, we aim to investigate waste clearance mechanisms in the human brain [1], [2].Therefore, we collect (longitudinal) data from multiple sources, in particular from Magnetic Resonance Imaging (MRI), ECG, SpO2, breathing belt, laboratory analysis of blood and urine.Our RDM System (RDMS) allows us to integrate these inhomogeneous data sources in one data base [3] where it is accessible via structured queries either via API or GUI.Furthermore, we developed (semi-) automatic post-processing pipelines that take care of routinely used post-processing steps.Computationally demanding tasks were set up to utilize high-performance computing (HPC) infrastructure, with automatic job submission and re-integration into the data base.Job submission can also be triggered via a GUI, which allows access to advanced, computationally demanding postprocessing tools for non-expert users.

Data catalogs are an established tool to integrate heterogeneous data, enrich raw data with semantic meaningful metadata, and make data easily searchable, maintainable, and shareable. This helps, for instance, data scientists to manage large data sets, which are often required for state-of-the-art artificial intelligence research. Driven by the increasing computing demand of these data-intensive projects, High-performance Computing (HPC) providers have to address the specific demands of these projects to attract this new user group to HPC systems. One particularly challenging domain are the life sciences working with highly-regulated, sensitive health data. This paper presents a workflow to deploy on-demand Elasticsearch (ES) clusters in user space on HPC systems, providing a backend for direct usage or user-defined, higher-order data catalog functionalities. Therewith, it augments the capabilities of a parallel file system by allowing processing of user-defined metadata. Two different encryption techniques are presented and used in two different use cases to systematically benchmark the developed setup, show its general scalability, and highlight important considerations when adapting it to a new use case. It is shown, that scaling out ES clusters has to be done using thorough data and workload modeling since larger clusters can be either beneficial or harmful to different workloads.

The increasing adoption of large language models (LLMs) has created a pressing need for an efficient, secure and private serving infrastructure, which allows researchers to run open-source or custom fine-tuned LLMs and ensures users that their data remains private and is not stored without their consent. While high-performance computing (HPC) systems equipped with state-of-the-art GPUs are well-suited for training LLMs, their batch scheduling paradigm is not designed to support real-time serving of AI applications. Cloud systems, on the other hand, are well suited for web services but commonly lack access to the computational power of clusters, especially expensive and scarce high-end GPUs, which are required for optimal inference speed. We propose an architecture with an implementation consisting of a web service that runs on a cloud VM with secure access to a scalable backend running a multitude of AI models on HPC systems. By offering a web service using our HPC infrastructure to host LLMs, we leverage the trusted environment of local universities and research centers to offer a private and secure alternative to commercial LLM services. Our solution natively integrates with Slurm, enabling seamless deployment on HPC clusters and is able to run side by side with regular Slurm workloads, while utilizing gaps in the schedule created by Slurm. In order to ensure the security of the HPC system, we use the SSH ForceCommand directive to construct a robust circuit breaker, which prevents successful attacks on the web-facing server from affecting the cluster. We have successfully deployed our system as a production service, and made the source code available at https://github.com/gwdg/chat-ai

The landscape of maintenance in distributed systems is rapidly evolving with the integration of Artificial Intelligence (AI). Also, as the complexity of computing continuum systems intensifies, the role of AI in predictive maintenance (Pd.M.) becomes increasingly pivotal. This paper presents a comprehensive survey of the current state of Pd.M. in the computing continuum, with a focus on the combination of scalable AI technologies. Recognizing the limitations of traditional maintenance practices in the face of increasingly complex and heterogenous computing continuum systems, the study explores how AI, especially machine learning and neural networks, is being used to enhance Pd.M. strategies. The survey encompasses a thorough review of existing literature, highlighting key advancements, methodologies, and case studies in the field. It critically examines the role of AI in improving prediction accuracy for system failures and in optimizing maintenance schedules, thereby contributing to reduced downtime and enhanced system longevity. By synthesizing findings from the latest advancements in the field, the article provides insights into the effectiveness and challenges of implementing AI-driven predictive maintenance. It underscores the evolution of maintenance practices in response to technological advancements and the growing complexity of computing continuum systems. The conclusions drawn from this survey are instrumental for practitioners and researchers in understanding the current landscape and future directions of Pd.M. in distributed systems. It emphasizes the need for continued research and development in this area, pointing towards a trend of more intelligent, efficient, and cost-effective maintenance solutions in the era of AI.

In the rapidly evolving fields of Artificial Intelligence (AI) and High-Performance Computing (HPC), benchmarking is a critical tool for optimizing system performance. It guides the selection of hardware architectures, software frameworks