Training Graph Neural Networks (GNNs) on large graphs is typically done on complex, distributed, or Out-of-Core (OOC) systems due to memory pressure. But we challenge this assumption as training typically requires access to only a small subset of node features. As a result, training on many large graphs exhibits relatively small working sets that can fit on a modest GPU. We present a simple single-GPU framework that pre-computes computation graphs and feature subsets to eliminate CPU–GPU data transfers and runtime overheads. Our approach achieves reductions in epoch time of up to 45× over the distributed DistDGL framework and ≈ 2× over the OOC DiskGNN framework, demonstrating that a modern GPU is all you need for efficient large-scale GNN training.
Accurate prediction of parallel application performance in HPC systems is essential for efficient resource allocation and system design. Classical performance models estimate of speedup based on theoretical assumptions, but their applicability is limited by parameter estimation, data acquisition, and real-world system issues such as latency and network congestion. This paper describes performance prediction using classical performance models boosted by a trainable machine learning framework. Domain-informed machine-learning models estimate the overhead of an application for a given problem size and resource configuration as a coefficient of the estimated speedup provided by performance laws. We evaluate this approach on two HPC mini-applications and two full applications with varying patterns of computation and communication and also evaluate the prediction accuracy on runs with varying processors-per-node configurations. Our results show that this method significantly improves the accuracy of performance predictions over standard analytical models and black-box regressors, while remaining robust even with limited training data.
The deployment of deep neural networks on edge devices remains challenging due to their computational demands and memory requirements. We introduce a novel compression framework that integrates Layer-wise Relevance Propagation (LRP) with high-dimensional clustering to simultaneously identify and eliminate both unimportant and redundant neurons. Unlike conventional magnitude or gradient-based pruning methods, our approach computes LRP scores across all network layers to capture functional dependencies, performs global k-means clustering to group functionally similar neurons, and implements parameter sharing through representative neurons or synthetic centroids. We provide empirical analysis demonstrating how relevance conservation principles yield more reliable importance estimates than traditional methods. Extensive experiments on MNIST and CIFAR-10 datasets using fully connected networks, ResNet-9, and ResNet-18 architectures demonstrate compression ratios up to 16x with minimal accuracy degradation (less than 2% in most cases), outperforming pruning methods including SNIP, GraSP, and Fisher pruning. Edge deployment on platforms ranging from NVIDIA Jetson to Raspberry Pi shows consistent performance and reduced inference latency, validating our method's practical applicability for resource-constrained environments.
We introduce a novel, efficient computational method, ZEUS, for numerical optimization, and provide an open-source implementation. It has four key ingredients: (1) particle swarm optimization (PSO), (2) the use of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) method, (3) automatic differentiation (AD), and (4) GPUs. Our approach addresses the computational challenges inherent in high-dimensional, non-convex optimization problems. In the first phase of the algorithm, we get a potentially good set of starting points using PSO. Thereafter, we run BFGS independently in parallel from these starting points. BFGS is one of the bestperforming algorithms for numerical optimization. However, it requires the gradient of the function being optimized. ZEUS integrates automatic differentiation into BFGS thus avoiding the need for the user to calculate derivatives explicitly. The use of GPUs allows ZEUS to speed up the calculations substantially. We carry out systematic studies to explore the trade-offs between the number of PSO iterations taken, starting points, and BFGS iteration depth. We show that a handful of iterations of PSO can improve global convergence when combined with BFGS. We also present performance studies using common test functions. The source code can be found at https://github.com/fnal-numerics/global-optimizer-gpu.
Efficient execution of range queries and exact queries on metadata is critical for many scientific workflows on large-scale high-performance computing (HPC) systems. Range queries in a distributed setting pose a challenge, as typical partitioning methods such as hash-based, list-based, and range-based partitioning do not work efficiently. There are also challenges in balancing the workload using locality-sensitive partitioning methods, which are capable of routing range queries to a subset of nodes to achieve efficiency. In this paper, we study prefix-hash tree and range-hash partitioning methods that fit the requirements of efficient range queries in distributed HPC systems. We also proposed a novel load-balancing method inspired by the Gossip protocol, which has similar complexity for most operations while keeping the coefficient of variation (CV) less than 7% across diverse metadata workload distributions, i.e., normal, uniform, and exponential. Our proposed methods outperform the commonly used Key-Value store, RocksDB, for exact queries by up to 15x and for range queries by up to 120,000x.
Meeting deadlines for data-intensive workflows on HPC systems is challenging as jobs experience varying wait times before resources become available. This impact is significant in hybrid HPC+Cloud scheduling, which can lead to resource idleness, deadline violations, and higher costs. To address these issues, we propose scheduling data-intensive workflows over a combined HPC+Cloud hybrid environment in a deterministic manner by scavenging unused HPC resources. We predict resource availability (RA) of HPC systems, and exploit this prediction to dynamically split resource allocation between HPC's unused and Cloud's on-demand resources to complete a workflow by a given deadline. The deterministic resource allocation allows for preloading input data for workflow tasks, avoiding execution delays. Further, we develop an adaptive scaling algorithm that effectively backs up the targeted HPC allocation on Cloud facilities to avoid workflow execution delays in the event of incorrect RA estimation. Experiments show that our scheduling technique imposes minimal impact on HPC production jobs, saves cost for > 75% workflow runs, suggests accurate budgets with a mean 7.11% to 14.75% cost estimation error, and finishes a mean 98% to 99.4% of tasks before deadlines.
This paper presents the Quantum-Inspired Error Detection Protocol (QIEDP), a novel error correction scheme for serial communication that adapts quantum stabilizer code principles-specifically multi-dimensional parity relationships and syndrome-based detection-to classical digital systems. QIEDP implements a (15,7,5) linear code providing 2-bit error correction with O(1) computational complexity using pre-computed lookup tables containing 105 correctable error patterns. Simulation results demonstrate consistent Bit Error Rate (BER) improvements of 1.6-1.8x over conventional UART with parity in typical IoT operating conditions (9-15 dB Signal-to-Noise Ratio(SNR)), despite syndrome collisions limiting double-bit correction to 85.7% efficiency. The protocol achieves 46.7% code efficiency with 100% single-bit and 85.7% double-bit error correction rates using only 360 bytes of memory, making it particularly suitable for resourceconstrained IoT applications requiring reliable communication in electromagnetically noisy environments.
Predicting application I/O time in large HPC clusters is a difficult task due to the shared nature of the resources involved and intermingled layers of the I/O stack. However, a more accurate prediction of application I/O time would improve resource scheduling and make it easier to identify scaling and portability bottlenecks. Currently, this type of prediction, which involves high-dimensional parameter spaces, is often performed by using machine learning, introducing another problem: data availability. Several related works require a large amount of data or access to the specific cluster's job scripts to perform the prediction, which can be a significant challenge for small and medium clusters or new systems without prior performance data. In this work, we use a transfer learning workflow and I/O modeling to create a targeted I/O time prediction tailored for a specific application. Our work demonstrates that we can utilize similar to 1% of the data required to create the base model prediction to achieve up to 87.25% of the predictions that fall within the acceptable range of I/O time variability.
Group convolutions are widely adopted in modern CNN architectures such as CondenseNet and ShuffleNet to enable efficient inference on GPUs. However, when the connection pattern between input and output channels does not exhibit regularity (unstructured group convolution), popular deep learning frameworks (e.g., PyTorch) often struggle with load balancing and data reuse issues leading to reduced performance. In this paper, we present a comprehensive optimization framework that combines a Knapsack-based partitioning approach with Integer Linear Programming (ILP) and advanced matrix reordering to optimize the deployment of unstructured group convolutions which are used in popular models such as CondenseNets that learn group connections. Specifically, we use knapsack algorithm to determine partition (group) sizes for the connections to minimize execution time and use an Integer Linear Programming (ILP) to assign connections to the partitions (group) output by the knapsack algorithm. We also employ three matrix reordering strategies-Hierarchical Clustering (HC), Iterative Clustering (IC), and Reverse Cuthill-McKee (RCM) on the matrix representing the input-output connectivity pattern to further improve the performance of our scheduling algorithm. Our experiments on ShuffleNet and CondenseNet demonstrate up to 1.9x speedups over PyTorch. Furthermore, augmenting ILP with reordering achieves an additional 1.3x improvement demonstrating the importance of optimizing for load balancing and data reuse.
With the increasing scale of High-Performance Computing (HPC) systems and a new awareness of the environmental impact of HPC, new strategies are required to improve the efficiency of resource usage on these systems. One such strategy is Dynamic Resource Management (DRM), which allows changing the resources assigned to a job dynamically during its execution. This increased flexibility in resource allocation and job scheduling can lead to improvements in several system efficiency metrics. Despite these benefits, DRM has not yet been established as a ready-to-use technology for production HPC systems. This is caused by the significant changes required in all the layers of the HPC system software stack, which are only achievable with an extensive and holistic co-design process between resource management software and applications. In this work, we demonstrate the applicability of a recently introduced, generic design approach for dynamic resources called Dynamic Processes with PSets (DPP), to enable DRM in realworld systems. To this end, we developed an exemplary, dynamic system software stack implementation following the DPP design principles throughout all layers. Based on this, we assess the applicability and performance of our approach using both synthetic benchmarks and job mixes consisting of several dynamic, real-world applications. On up to 100 nodes, we measure moderate overheads for process reconfiguration in applications while significantly improving the system throughput and average job turnaround time compared to static scheduling in crowded system scenarios.
Non-uniform all-to-all communication patterns present optimization challenges in parallel computing due to their irregular data distribution and dynamic behavior. While MPI_Alltoallv provides the standard interface for such exchanges, achieving optimal performance requires careful selection among multiple implementation variants and tuning of algorithm-specific parameters. This paper presents a data-driven autotuning framework that combines machine learning-based runtime prediction with a lookup-table mechanism for fast configuration selection. The ML model estimates the communication time of each algorithm configuration under a given system setup, allowing the framework to identify the optimal implementation and parameter set based on predicted performance. We validate our approach through comprehensive benchmarking of MPI_Alltoallv and two specialized algorithms across varying process counts, message sizes, and tunable parameters. Applied to a real MPI-based transitive closure application on the Fugaku supercomputer, our framework achieves up to 6.03x reduction in communication time over the vendor implementation, providing a detailed understanding of non-uniform collective communication behavior and a practical framework for automatic performance optimization in HPC applications.
In modern multi-/many-core HPC systems, the increasing demands for data transfers and memory I/O workloads have emerged as a predominant challenge for parallel programming models like MPI. Specifically, memory copy performance within the node has emerged as a significant bottleneck of the existing state-of-the-art MPI intra-node communication scheme. This paper introduces a hybrid approach to offload memory copy tasks partly from CPU, utilizing the Linux DMA Engine API with support for various CPU architectures, including I/OAT and PTDMA. We characterize the performance of DMA channel-based copy design for further optimizations. Subsequently, we integrate the DMA memory copy design into the existing MPI intra-node communication framework, thereby empowering the MPI intra-node point-to-point communication to simultaneously leverage DMA-based schemes and existing CPU-driven schemes in a cooperative manner. To demonstrate the effectiveness of our proposed design, we conducted systematic experiments on two emerging architectures. Our experimental results demonstrate up to 26 % lower communication latency in microbenchmarks, up to 23 % faster execution time in the 3D-stencil application.
Maximizing the performance of deep learning models on AI accelerators like Tenstorrent Wormhole requires precise control over hardware resources such as compute cores and onchip memory. Operations such as convolution expose a range of tunable parameters, such as parallelization strategies, buffer sizes, compute datatypes, and memory hierarchies (SRAM vs. DRAM) that involve trade-offs between performance, memory usage, and numerical accuracy. Tenstorrent's TT-NN API reimplements PyTorch's conv2d while exposing all these lowlevel controls, allowing fine-grained control, but presenting a steep learning curve for users familiar with PyTorch's high-level abstractions. We propose a predictive software layer that facilitates the execution of conv2d on Tenstorrent hardware by modeling the relationship between input tensors, configuration parameters, and execution outcomes. We train two machine learning models: one predicts execution success with 99.82% accuracy, and the other estimates run-time performance with an R-2 score of 0.9984. These models enable automated hardware-aware tuning (TuneNet) of the TT-NN configuration space, reducing trial and error, avoiding out-of-memory errors, and improving performance. TuneNetselected configurations reduce conv2d latency by 13.29% compared to TT-NN defaults on Tenstorrent Wormhole N150s, while incurring minimal computational overhead (9s). This work introduces the first hardware-aware autotuning pipeline for Tenstorrent accelerators, significantly reducing tuning overhead while improving performance.
Data sizes are growing rapidly, leading to large document sizes and higher processing times. With the unprecedented growth in structured and semi-structured data, and their processing in various applications, XML continues to be a preferred choice as a standard data format and exchange. This standardization permits vendors to build tools for parsing and querying XML data. Unfortunately, the absolute running times of these tools quickly become prohibitive for large XML data. In such a situation, parallel processing becomes imperative. However, parsing and querying XML documents in parallel present significant challenges. Parsing is particularly difficult due to the inherent sequential and hierarchical nature of XML data, while complex querying with regular expressions is constrained by memory limitations. In this work, we develop a parallel XML parser and a parallel XML query engine with support for regular expressions. It supports not only the traditional startswith and endswith XPath queries, but also presence of arbitrary Kleene-closures. An extensive experimental evaluation of our tool illustrates significant improvements over standard libXML, especially at scale, achieving over 32x speedup.
Nowadays, HPC faces several changes, both from applications and hardware heterogeneity. However, most HPC resource managers still present a centralized and rigid approach. The application of HPC in new scientific fields has led to a proliferation in the execution of scientific workflows, usually composed of a large number of tasks and dependencies between them. Thus, the exploitation of task-level parallelism has become paramount for resource utilization in current HPC, a growing concern along with power consumption. Due to their traditionally centralized and rigid scheme, they require a re-evaluation to address these new challenges, such as the previously mentioned workflow scheduling or malleability. Malleability appears as a promising solution to increase system throughput, allowing the jobs to change resources at runtime. Pilot-Jobs offer a solution to the rigidity of current HPC resource managers, allowing them to join all the necessary resources required for a workflow execution in one job and then perform fine-grained scheduling on those resources. This work presents a decentralized and user-level task manager based on tuple spaces and the LINDA communication model within a Pilot-Job. This approach allowed us to leverage finegrained scheduling, exploiting task parallelism in the different phases that compose our workflows. In a real deployment within the operational marine monitoring service IBI-MFC provided by the European Commission through the Copernicus Project, reducing execution time and energy consumption by 40% and 30%, respectively. This service is maintained by Nologin Oceanic Weather Systems S.L.U. and uses the HPC resources of the Galician Supercomputing Center (CESGA) in Santiago de Compostela, Spain. Also, we implemented a mechanism to allow core-level malleability, enabling our system to modify the cores assigned to each task at runtime, achieving up to 10% time reduction.
We develop a distributed-memory algorithm to embed nodes of a graph into a low-dimensional vector space. Our distributed algorithm, called DistFNE, is based on a force-directed layout that maximizes attraction among neighboring vertices and repulsion among distant ones. DistFNE utilizes large minibatches in stochastic gradient descent (SGD) to scale node embeddings to thousands of processors without compromising the quality of the embeddings. DistFNE optimizes memory usage and inter-process communication through a configurable push-pull strategy. We employ asynchronous MPI communication to overlap communication and computation. These combined techniques made DistFNE scalable to 32,768 cores of supercomputers to generate embeddings of graphs with billions of edges.
Graph Neural Networks (GNNs) are increasingly used in domains such as finance and bioinformatics, where both node features and edge structures can contain sensitive information. While Fully Homomorphic Encryption (FHE) offers a promising solution for privacy-preserving GNN inference, existing approaches such as PPGNN rely on costly Homomorphic Rotation and MUX operations for operand obfuscation, resulting in significant computational overhead. In this work, we propose a new obfuscation method that leverages the probabilistic nature of FHE to duplicate ciphertexts at the client side, thereby eliminating the need for runtime selection logic. To support this method efficiently, we design a pipelined hardware accelerator with a simplified CKKS datapath and parallel TFHE execution, avoiding the complexity of rotation-heavy designs. Despite reduced ciphertext reuse, our architecture mitigates memory pressure through buffer-aware PBS unit design. Experimental results demonstrate up to 8.8x speedup and 7.69x energy efficiency improvement over PPGNN, while also outperforming existing multi-scheme accelerators such as Trinity and UFC even when applying the same obfuscation strategy. Our approach offers a practical and scalable solution for efficient, privacy-preserving GNN inference.
The deployment of Large Language Models (LLMs) is increasingly challenged by their substantial computational and memory demands, straining the capabilities of current hardware. A significant, yet often less explored contributor to this inefficiency lies within the Multi-Layer Perceptron (MLP) blocks of transformer architectures, where sequences of tensor operations introduce considerable overhead. Although attention mechanisms have been the focus of extensive optimization, the MLP component has remained a comparatively underexplored frontier for performance enhancement. Our paper aims to address this critical gap in LLM inference by introducing Iterative MLP Blocks with Parameter Splits (IMBPS). IMBPS introduces a lossless blocking strategy designed to enhance LLM inference efficiency through two principal mechanisms. We present a concise analytical model that facilitates effective mitigation of memory-related overhead, specifically targeting data-packing and peak activation footprints in MLP computations. This approach leads to a significant reduction in cache misses. Our empirical evaluations provide compelling evidence of these benefits, with results indicating a 1.5-1.6x speedup for standalone MLP blocks and end-to-end (E2E) performance improvements of up to 1.37x.
Nowadays, due to the increasing number of cores and sockets in modern multiprocessor servers, it is essential to coschedule multiple applications simultaneously to maximize system utilization. However, the performance and energy efficiency of coexecuting applications are highly sensitive to thread placement, core allocation, and both core and uncore frequency settings. Existing dynamic resource allocation solutions often rely on model-based approaches, require intrusive modifications to the parallel runtime, or lack a unified framework that can utilize multiple energy-saving mechanisms while supporting different parallel programming models. This paper presents Harmonizer, a novel dynamic resource optimization library for co-running applications on multiprocessor systems to improve overall system throughput and energy efficiency. Harmonizer is oblivious to the parallel programming model and requires no information from prior executions. It periodically profiles each application's CPU, cache, and memory usage by utilizing hardware performance monitoring counters and uses this data to determine optimal thread placement, core allocation, and frequency settings. We evaluate Harmonizer using several co-running mixes of exascale proxy applications on a four-socket, 72-core Intel Cooper Lake processor. Our results show that Harmonizer reduces energy consumption by 8.8% to 35% (20.5% geometric mean) and improves throughput by 4.8% (geometric mean) compared to the default Linux scheduler. Compared to two state-of-the-art approaches, it achieves up to 28.6% energy savings and 15.8% higher throughput (geometric mean).
Python's ease of use has driven its adoption in data science, machine learning, and increasingly, high-performance computing (HPC), but its performance lags due to its dynamic nature. While many efforts accelerate Python by restricting its features, this work leverages Python's dynamism to accelerate it. We introduce control state serialization to Python through Sauerkraut, a library that captures the complete execution state (call stack, instruction pointers, operand stacks, local variables, global context) of running functions, complementing existing data serialization. Sauerkraut enables snapshots of function execution to be serialized, transferred, and resumed later or elsewhere. Sauerkraut is compatible with off-the-shelf Python installations. We demonstrate its utility by building Kombucha, a partial MPI implementation that provides general-purpose load balancing by migrating virtualized MPI ranks between processes using Sauerkraut to transfer control state. Evaluations show Sauerkraut adds minimal overhead over standard serialization, and Kombucha achieves significant speedups (up to 2.21x for a CPU Particle-in-Cell code and 1.43x for a GPU Jacobi3D code with synthetic imbalance) with minimal application code changes. Control state serialization opens new avenues for performance optimization in Python, including load balancing, checkpoint/restart, and replay debugging, enhancing Python's suitability for the HPC community.
