Halo exchange is the task of communicating the boundaries of a subdivided computational grid with neighboring processes in distributed computations. With graphics processors augmenting the world’s fastest supercomputers, data movement across heterogeneous memory hierarchies must be carefully orchestrated to leverage these accelerators at scale. In this work, we implement and optimize packing, inter-process communication, domain decomposition, and topology-aware process assignment integral to efficient halo exchange on heterogeneous systems using GPU-aware MPI. Our test cases include benchmarks of the halo exchange on one- to five-dimensional grids on both AMD and Nvidia graphics processors. We find that in multidimensional halo exchange, custom packing kernels provide up to two orders of magnitude speedup compared to MPICH and OpenMPI implementations based on MPI datatypes. Furthermore, hierarchical topology-aware process assignment provides more consistent performance across varying grid sizes on heterogeneous systems compared to simpler methods. We apply our methods improve the strong scaling of magnetohydrodynamics simulations, demonstrating 1.58× speedup on high process counts in parallel computation and communication. Our results highlight the importance of kernel fusion for alleviating small kernel overheads on graphics processors even if it prohibits pipelining packing with communication.
Achieving peak performance for ocean models on modern high-performance computing (HPC) architectures requires extensive, costly code rewrites that are not only time-consuming and error-prone but also highly architecture-specific and require numerics experts to be proficient in parallel programming models or domain-specific languages (DSLs). In this paper we introduce Poseidon , a source-to-source code optimization tool that employs numerics & HPC co-design. We developed an uplifting approach to a hypergraph for the data flow and to our own Poseidon Intermediate Representation (PosIR) for the computations, which recovers for the first time high-level information and semantics about the computations and memory management that are typically lost during the conversion of numerical algorithms to source code. This representation is then employed for model-driven optimizations. In the backend, we inject source code back into the original code supporting different target HPC architectures, reusing the existing model as the runtime. Our evaluation investigates OpenMP and OpenACC parallel programming models on CPUs and GPUs. We demonstrate Poseidon’s capability of automatic parallelization and optimization of existing Fortran code, with comparisons to state-of-the-art parallelized code with performance improvements of up to × 2.60 and automatically reducing the required memory footprint by up to × 2.29.
Fully-resolved micromechanical simulations coupling the Lattice Boltzmann Method (LBM) with the Discrete Element Method (DEM) provide high-fidelity insights into granular fluidization. However, the substantial computational demands of such simulations require efficient implementations on supercomputing architectures. This work presents a comprehensive performance analysis of a fully-resolved LBM-DEM model to study granular fluidization, i.e., piping erosion, during the installation of an offshore caisson foundation. The performance evaluation focuses on real-world workloads rather than simplified benchmark problems to obtain realistic performance insights. The study considers a diverse range of state-of-the-art HPC hardware architectures, namely the LUMI and MareNostrum 5 EuroHPC supercomputers, both CPU and GPU partitions. The results demonstrate that GPU-based systems generally outperform the CPU-based systems. Strong and weak scaling analyses were conducted with up to 512 nodes, and parallel efficiencies reached up to 92%. Nonetheless, the results also indicate that atomic add operations on GPUs can become a bottleneck for the parallel efficiency at large scales. Moreover, the study reveals a close link between physical variations of the setup and significant scaling implications, underscoring the need to consider physical model characteristics in scaling assessments.
Simulating large-scale, high-fidelity population health models demands immense computational resources and efficient parallelization strategies. We present the design and performance of Enable (Efficient National-scale Agent-Based Learning Environment), a hybrid CPU-GPU agent-based modeling framework optimized for the Frontier supercomputer. Enable generates contact networks directly from activity schedules, enabling location-based parallel execution without relying on precomputed contact graphs. A GPU device constructs contact networks and uses an efficient load balancing algorithm to assign locations to processors, while CPUs perform the simulation tasks. This design achieves high parallel efficiency by ensuring balanced edge distributions across processors, resulting in uniform execution times. We evaluate both strong and weak scaling using synthetic and real-world datasets generated from UrbanPop, Uber H3, and OSM maps. Scaling performance is studied with city-to national-scale population sizes on up to 1200 GPUs of the Frontier supercomputer. Enable addresses key computational challenges in large-scale high-fidelity agent-based simulations that beset the development of national-scale virtual population health twins.
Large-scale Earth observation (EO) datasets are crucial for identifying environmental changes, particularly in coastal regions vulnerable to erosion. However, analyzing these massive datasets requires computational techniques beyond the capabilities of conventional workstations or single nodes of an HPC system. In this study, we demonstrate how Heat, a highly parallel, open-source, Python-based library designed for scalable machine learning and data processing, can address this issue-quasi as an off-the-shelf solution for up-scaling-when detecting anomalies in satellite imagery of the German North Sea coast. Our implementation of an unsupervised outlier-based anomaly detection algorithm uses Heat's distributed arrays and vectorized map (vmap) primitives. This algorithm processes tens of millions of shoreline locations by sharing memory and computation across multi-node CPU/GPU clusters, with minimal code changes compared to NumPy/PyTorch. Weak-scaling experiments on both CPUs and GPUs demonstrate the scalability of our approach with increasing amounts of data. We thereby provide, to the best of our knowledge, the first density-based anomaly detection on large-scale EO datasets in a multi-node setting, with proven portability to different hardware architectures (x86-and ARM-based CPUs) and vendors (Nvidia and AMD GPUs). The resulting anomaly maps align with known severe weather episodes, daily anomaly counts correlate with wind metrics from coastal stations, and hotspot maps identify regions of high activity. These findings support the geophysical plausibility of the detections. Our approach is reproducible due to its deterministic algorithms, extensible to additional EO modalities (e.g. coherence), and broadly applicable to nationwide monitoring. Although the 20 m data resolution limits detection sensitivity, the method itself is resolution-agnostic.
This study introduces a novel framework designed to enhance the performance, scalability, and portability of the kilometer-scale E3SM Land Model (km-ELM) within the E3SM modeling infrastructure. By seamlessly integrating cutting-edge data tools, we address existing challenges such as slow performance, limited scalability, and difficulties in software integration in current data-driven ELM simulation over large geographic areas. Our innovative approach leverages the KiloCraft data toolkit to generate unified inputs for simulations ranging from a single-cite case, to a 72,083-cell regional case to a continental configuration encompassing 21.6 million land grid cells at a 1 km & times; 1 km resolution. We conduct extensive strong- and weak-scaling experiments on three state-of-the-art supercomputers, utilizing up to 100,800 CPU cores across 2400 compute nodes to evaluate end-to-end metrics including wall-clock time, simulation-years-per-day (SYPD), initialization costs, and I/O throughput. Our results reveal the land (LND) component's efficient scaling, demonstrating near-ideal weak scaling and strong-scaling parallel efficiencies reaching up to 87% at 50,400 cores. We confirm portability and reproducibility through bitwise-equivalent outputs across different machines using identical inputs over supported machines. Notably, at extreme scales, we identify I/O as a critical bottleneck and that leads to effective solution with the SCORPIO/ADIOS stack. Collectively, these findings validate the deployment of km-ELM at a continental scale with high parallel efficiency and provide essential guidance on configuration, decomposition, and I/O settings for optimized kilometer-scale land simulations in E3SM. This work emphasizes the innovative design and practical solutions that enhance the operational capabilities of km-ELM, focusing on software performance and scalability while leaving detailed scientific evaluations of simulated land processes for future investigations.
This paper explores highly efficient cooperative strategies for population-based parallel metaheuristics, focusing on the integration of shared memory and Remote Memory Access (RMA) operations provided by MPI-3. Most parallel metaheuristic proposals use island-based models with point-to-point communications in their cooperative strategies. These communications can saturate the network and buffers by sending information that often will not be used at the destination, thus resulting in a waste of resources when dealing with these types of applications. In this paper we evaluate other alternative communication protocols that use shared memory windows and RMA operations. Under these new models, the movement of information over the network is on-demand and requested by the source, maximizing resource efficiency. Although the approaches and descriptions in the paper are generic, a particular metaheuristic, the Ant Colony Optimization (ACO), is used to carry out the experiments. The results obtained draw interesting conclusions that can serve to guide the future parallelization of other metaheuristics.
Scientific high performance computing (HPC) applications employ asynchronous executions of MPI calls to improve scalability and performance. The asynchronous calls can lead to non-determinism (ND) in execution, particularly for large exascale simulations. In order to ensure reproducibility and facilitate error detection, it is imperative to identify the sources of non-determinism. Message ND that occurs when the order in which a process sends or receives MPI communication, or executes MPI calls varies across different runs of the same application. We model the MPI calls in the execution as an event graph. The regions of dissimilarity between two event graphs indicate the sources of non-determinism in the MPI calls. Thus by aligning the nodes of the event graphs, we can identify sources of ND. We show that traditional alignment techniques such as NetAlign and learning methodologies such as Graph Autoencoders are not able to align graphs with high accuracy due to the nearly regular degree and large diameter of event graphs. Therefore, we propose a meta graph heuristic that exploits structural properties of event graphs, by combining the set of nodes representing sequences of MPI calls within the same processor as a meta node. We align the meta graphs formed from these meta nodes, and then align the individual nodes within the meta nodes. Our results over three different MPI applications highlight that our meta graph heuristic has better accuracy and scales to large graphs compared to network alignment and graph auto encder methods.
We describe chipStar, an open source software stack that enables building unmodified CUDA and HIP programs into binaries that rely solely on open cross-vendor compute standards OpenCL and SPIR-V. The relevant technical aspects of chipStar and the feature mismatches between the CUDA/HIP APIs and OpenCL are discussed along with a set of standard extension proposals to bridge the essential gaps in the future. The key benefit of the software stack is its portability, which is demonstrated by providing performance evaluations on a diversity of less common CPU/GPU platforms including RISC-V/PowerVR and ARM Mali. A comparison against the original AMD HIP platform provides a geometric mean of 0.75, a reasonable price to pay for the enhanced portability. chipStar is now considered mature enough for wider testing and even production use, which is demonstrated by successful porting and competitive performance of GAMESS-GPU-HF, a complex HPC application.
In Part I of this paper (Hanindhito et al., 2026), we argued understanding technology trends in computing hardware is necessary for designing next-generation algorithms for scientific computing. Using a language that is accessible to a general computational or data scientist, or applied mathematician, we aim to help our targeted readers better understand how technology trends in computing hardware are going to impact their computations, and what characteristics algorithms should exhibit to best harness modern hardware. In Part I, we covered background material, general-purpose processors, and hardware accelerators. In Part II, we review memory systems, inter-device communication, heterogeneous computing and system integration, and why energy efficiency has become a central issue in hardware design, and why it cannot be ignored in cluster-sized computers. We conjecture how the above changes may impact scientific computing, and offer options for leveraging modern hardware for both old and modern scientific software.
Plants play a vital role in living organisms. However, daily changes in the environment can significantly impact plant growth with respect to their surroundings. Exposure to pathogens, environmental disturbances, or unfavorable conditions make plants vulnerable to diseases. The rapid spread of diseases affects the healthy growth of plants, resulting in the reduction of products quality and quantity. To mitigate these issues, it is necessary to monitor the plant growth and to identify the plant diseases at an earlier stage. However, the manual detection of plant diseases is an unreliable method as it is a time-consuming and error-prone process. Analyzing the subjected plant leaves through the advancements in image processing has paved the way to engage the researchers in the development of automated solutions for disease identification. In addition, image acquisition, data collection, and the development of advanced state-of-the-art approaches are the major research tools contributed by various researchers. Therefore, this paper delves into the latest methodologies by concentrating on these research contributions. In this regards, the available research papers published from 2018 to 2024 are considered. This study also incorporates some of the challenges and research topics for future plant leaf disease analysis by emphasizing the critical role of high-performance computing. Moreover, it offers valuable insights for researchers and practitioners, providing a wide-ranging understanding of the current state-of-the art approaches in the field of plant leaf disease analysis by highlighting their pros and cons.
One of the most important aspects of supercomputer development in the post-Moore era is the interconnect technologies that allow one to unite a multitude of processing elements into a well-synchronized computing system. Novel types of supercomputer interconnect require careful benchmarking and compliance with the requirements of modern hardware trends. GPU-based heterogeneous computing is one of the most important current avenues for building high performance computing systems, and the support of GPU-aware MPI technology is a requirement for any competitive interconnect. In this paper, we describe a UCX API based GPU-aware MPI implementation for the Angara interconnect. Performance analysis for peer-to-peer, MPI_Bcast and MPI_Reduce operations is presented, as well as for the rocHPL benchmark and for a typical biomolecular model within the LAMMPS molecular dynamics code. The deployment of the Desmos supercomputer equipped with both Angara and InfiniBand FDR allows us to make an accurate comparison of these two types of interconnect using the latter as a reference.
Efficient simulation of complex plasma dynamics is crucial for advancing fusion energy research. Particle-in-Cell (PIC) Monte Carlo (MC) simulations provide insights into plasma behavior, including turbulence and confinement, which are essential for optimizing fusion reactor performance. Transitioning to exascale simulations introduces significant challenges, with traditional file input/output (I/O) inefficiencies remaining a key bottleneck. This work advances BIT1, an electrostatic PIC MC code, by improving the particle mover with OpenMP task-based parallelism, integrating the openPMD streaming API, and enabling in-memory data streaming with the ADIOS2 Sustainable Staging Transport (SST) engine to enhance I/O performance, computational efficiency, and system storage utilization. We employ profiling tools such as gprof, perf, IPM and Darshan, which provide insights into computation, communication, and I/O operations. We implement time-dependent data checkpointing with the openPMD API enabling seamless data movement and in-situ visualization for real-time analysis without interrupting the simulation. We demonstrate improvements in simulation runtime, data accessibility and real-time insights by comparing traditional file I/O with the ADIOS2 BP4 and SST backends. The proposed hybrid BIT1 openPMD SST enhancement introduces a new paradigm for real-time scientific discovery in plasma simulations, enabling faster insights and more efficient use of exascale computing resources.
We present a new parallel spectral element solver, FNPF-SEM, for simulating linear and fully nonlinear potential-flow-based water waves and their interactions with offshore structures. The tool is designed as a general-purpose wave model for offshore engineering applications. Built on the open-source Firedrake framework, the new FNPF-SEM model is a computational tool capable of capturing both linear and nonlinear wave phenomena with high accuracy and efficiency, supporting high-order (spectral) finite elements. Additionally, Firedrake provides native support for MPI-based parallelism, enabling efficient multi-CPU distributed computations for large-scale simulations. We demonstrate the capabilities of the high-order model through h- and p-convergence studies, and weak and strong scaling tests. Verification is performed against analytical and numerical solutions and verified against experimental data, including nonlinear high-order harmonic generation and wave interactions with a cylinder and a breakwater. The new FNPF-SEM model offers a numerical framework for simulating wave propagation and wave-structure interactions, with the following key features: (i) the ability to represent complex geometries through flexible, unstructured finite element meshes; (ii) reduced numerical diffusion and dispersion by using high-order polynomial expansions; and (iii) scalability to full- and large-scale simulations over long time periods through a parallel implementation.
This paper addresses emulation algorithms for matrix multiplication. General Matrix-Matrix Multiplication (GEMM), a fundamental operation in the Basic Linear Algebra Subprograms (BLAS), is typically optimized for specific hardware architectures. The Ozaki scheme is a well-established GEMM-based emulation method for matrix multiplication, wherein input matrices are decomposed into several low-precision components to ensure that the resulting matrix product is computed exactly through numerical operations. This study proposes a novel GEMM-based emulation method for matrix multiplication that leverages the Chinese Remainder Theorem. The proposed method inherits the computational efficiency of highly optimized GEMM routines and further enables control over the number of matrix multiplications, which can enhance computational accuracy. We present numerical experiments featuring INT8 Tensor Core operations on GPUs and FP64 arithmetic on CPUs as case studies. The results demonstrate that FP64 emulation using the proposed method achieves performance levels of up to 7.4 to 9.8 TFLOPS on the NVIDIA RTX 4090 and 56.6 to 80.2 TFLOPS on the NVIDIA GH200, exceeding the measured performance of native FP64 arithmetic. Furthermore, for FP64 computations on CPUs, the proposed method achieved up to a 2.3x speedup in emulating quadruple-precision arithmetic compared to the conventional Ozaki scheme.
Recent advancements have positioned AI, and particularly Large Language Models (LLMs), as transformative tools for scientific research, capable of addressing complex tasks that require reasoning, problem-solving, and decision-making. Their exceptional capabilities suggest their potential as scientific research assistants but also highlight the need for holistic, rigorous, and domain-specific evaluation to assess effectiveness in real-world scientific applications. This paper describes a multifaceted methodology for Evaluating AI models as scientific Research Assistants (EAIRA) developed at Argonne National Laboratory. This methodology incorporates four primary classes of evaluations. 1) Multiple Choice Questions to assess factual recall; 2) Open Response to evaluate advanced reasoning and problem-solving skills; 3) Lab-Style Experiments involving detailed analysis of capabilities as research assistants in controlled environments; and 4) Field-Style Experiments to capture researcher-LLM interactions at scale in a wide range of scientific domains and applications. These complementary methods enable a comprehensive analysis of LLM strengths and weaknesses with respect to their scientific knowledge, reasoning abilities, and adaptability. Recognizing the rapid pace of LLM advancements, we designed the methodology to evolve and adapt so as to ensure its continued relevance and applicability. This paper describes the methodology state at the end of February 2025. Although developed within a subset of scientific domains, the methodology is designed to be generalizable to a wide range of scientific domains.
With heterogeneous systems, the number of GPUs per chip increases to provide computational capabilities for solving science at a nanoscopic scale. However, low utilization for single GPUs defies the need to invest more money in expensive accelerators. Although related work develops optimizations to improve application performance, none studies how these optimizations impact hardware resource usage or average GPU utilization. This paper takes a data-driven analysis approach in addressing this gap by (1) characterizing how hardware resource usage affects device utilization, execution time, or both, (2) presenting a multiobjective metric to identify important application-device interactions that can be optimized to improve device utilization and application performance jointly, (3) studying hardware resource usage behaviors of several optimizations for a benchmark application, and finally (4) identifying optimization opportunities for several scientific proxy applications based on their hardware resource usage behaviors. Furthermore, we demonstrate the applicability of our methodology by applying the identified optimizations to a proxy application, which improves the execution time, device utilization, and power consumption by up to 29.6%, 5.3% and 26.5% respectively.
We implement and analyse a sparse/indirect-addressing data structure for the Lattice Boltzmann Method to support efficient compute kernels for fluid dynamics problems with a high number of non-fluid nodes in the domain, as encountered in porous media flows. The data structure is integrated into a code generation pipeline to enable sparse Lattice Boltzmann Methods with a variety of stencils and collision operators and to generate efficient code for kernels on CPU as well as on AMD and NVIDIA accelerator cards. To further enhance performance, we optimize these sparse kernels with an in-place streaming pattern to save memory accesses and memory consumption and we implement a communication hiding technique to demonstrate strong scalability. We provide a comprehensive, systematic performance analysis comparing the sparse and the traditional dense data structure. We present single GPU performance results for the sparse approach with up to 99% of maximal bandwidth utilization. We integrate the optimized generated kernels in the high performance framework waLBerla and achieve a scaling efficiency of at least 82% on up to 1024 NVIDIA A100 GPUs and up to 4096 AMD MI250X GPUs on modern HPC systems. In addition, we propose a hybrid data structure that enables an adaptive choice between sparse and dense representations on a per-subdomain basis, allowing further improvements in performance and memory efficiency. We evaluate all approaches on three realistic application scenarios: flow through porous media, free flow over a particle bed, and blood flow in a coronary artery. Across these benchmarks, we demonstrate a speed-up of up to 2x and a reduction in memory consumption of up to 75% using the sparse/indirect-addressing data structure compared to the conventional direct-addressing approach.
