The diversity of communication paths in a network—especially non-minimal paths—is a key enabler of performance at extreme scales. We present EvalNet, a toolchain for scalable generation and analysis of over 25 important network topologies, such as Slim Fly, PolarFly, and Orthogonal Fat Trees, with a strong focus on path diversity metrics. EvalNet provides an extensive and fine-grained analysis of shortest and non-shortest paths, including their multiplicities, lengths, and interference. It supports exact measurement and visualization of bandwidth and throughput between every router pair, enabling unprecedented insight into routing potential. EvalNet also includes detailed models for construction cost and power consumption, and interfaces seamlessly with established simulators, which we tune to support large-scale evaluations on low-cost hardware. Using EvalNet, we deliver the widest and most comprehensive path diversity study to date, demonstrating how path diversity underpins throughput and scalability, and facilitating progress towards new frontiers in extreme-scale network design.Code: https://github.com/spcl/EvalNetExtended technical report: https://arxiv.org/abs/2105.12663
Synchronization mechanisms such as locks and condition variables significantly impact the performance of parallel applications. While numerous algorithms have been proposed to implement these mechanisms, most rely on either thread spinning or system calls to manage waiting queues. Both approaches introduce inefficiencies: thread spinning leads to excessive CPU usage, while system calls incur costly context switches between user and kernel spaces, particularly in contended scenarios.This paper presents QFSync, a novel queue-free synchronization approach aimed at reducing the number of system calls and eliminating unnecessary thread spinning. When a thread requires blocking, it is suspended for a short time interval using the clock_nanosleep system call. The thread is then awakened directly by a clock interrupt when the interval elapses.We propose two algorithms for implementing POSIX mutex locks and condition variables using this technique. Extensive evaluations using micro-benchmarks and parallel applications demonstrate that QFSync outperforms traditional POSIX implementations by significantly reducing the number of system calls.
We describe two adaptive sorting paradigms, organic mergesort and finger buffer-tree sort, which have parallel or external-memory implementations. Depending on the implementation regime, our mergesort algorithms have optimal time, work, and/or input/output (I/O) operations with respect to the run-based entropy of the input sequence, which matches the performance of optimal sequential internal-memory natural mergesort algorithms, such as Timsort, Peeksort, and Powersort.
Minecraft is one of the most popular massively multiplayer environments, supported by an active and open community that continuously works to improve it. Among these contributions, MultiPaper focuses on addressing a critical challenge: the limited scalability of standard Minecraft servers, which restricts the number of users that can interact within a single shared world. Despite its innovations, our novel evaluation of MultiPaper reveals scaling limitations to maintain a good experience for large user counts.To address this, we introduce MetaPaper, an extension of MultiPaper that presents novel advancements in Minecraft’s multiplayer scalability. We present two key innovations: dynamic scaling, which enables elastic infrastructure and automated server deployment; and dynamic load balancing, allowing seamless player distribution and real-time migration across server instances. Our empirical analysis demonstrates that MetaPaper significantly improves both user experience and system performance, increasing playability and supporting substantially more concurrent users within the same virtual space (from 250 to 800 users).
Many recent efforts in developing hardware-accelerated spiking neural networks (SNNs) are characterized by deep co-design between algorithms, architectures, and devices. Architectural advances overcome device constraints by coupling together many small resistive-RAM (ReRAM) crossbars via a network-on-chip (NoC) for neuromorphic component operation. Concurrently, improved SNN training methods increase accuracy and structural sparsity in networks despite growing problem sizes. Finally, compilers leverage these attributes to minimize area and inter-crossbar communication while mapping large SNNs to sophisticated architectures. However, for compiler-driven co-design to realize increasingly complex and profitable optimizations, a compile-time view of power consumption is critical. We present PowerMappeR to express and optimize over mapping-, architecture-, and device-specific power consumption information. By modeling the dynamic power of well-established components, we develop an integer linear programming (ILP)-based, encoding-agnostic, parametric power estimation model. Using this model, we demonstrate practical improvements in area and inter-crossbar communication by 0%–9.5% and 1.4%–5.1%, respectively. We also limit hotspot formation during optimization, achieving comparable or better results in targeted metrics with up to 96.4%–97.1% restriction of hotspot magnitude. Finally, we introduce profile-guided formulations to reduce worst-case and expected-case hotspot magnitude by 40.7%–69.5% and 40.6%–56.3%, respectively. Optimizing worst-case hotspot magnitude incidentally improves expected-case magnitude by 10.85%–33.45%. Reciprocally, optimizing expected-case magnitude incidentally improves worst-case magnitude by 4.33%–39.87%. Validation against hardware simulators confirms that PowerMappeR can decrease dynamic power consumption by 12.6%–27.3%.
Evaluating a query over a large, irregular graph is inherently challenging. This challenge intensifies when solving a query over a sequence of snapshots of an evolving graph, where changes occur through the addition and deletion of edges. We carried out a study that shows that due to the gradually changing nature of evolving graphs, when a vertex-specific query (e.g., SSSP) is evaluated over a sequence of 25 to 100 snapshots, for 67.8% to 99.8% of vertices, the query results remain unchanged across all snapshots. Therefore, the Unchanged Vertex Values (UVVs) can be computed once and then minimal analysis can be performed for each snapshot to obtain the results for the remaining vertices in that snapshot. We develop a novel intersection-union analysis that accurately computes lower and upper bounds of vertex values across all snapshots. When the lower and upper bounds for a vertex are found to be equal, we can safely conclude that the value found for the vertex remains the same across all snapshots. Therefore, the rest of our query evaluation is limited to computing values across snapshots for vertices whose bounds do not match. We optimize this latter step evaluation by concurrently performing incremental computations on all snapshots over a significantly smaller subgraph. Our experiments with several benchmarks and graphs show that we need to carry out per snapshot incremental analysis for under 42% of vertices on a graph with under 32% of edges. Our approach delivers speedups of 2.01-12.23× compared to the state-of-the-art RisGraph implementation of the KickStarter-based incremental algorithm for 64 snapshots.
Attention mechanisms are crucial for enabling large language models (LLMs) to process long-sequence inference tasks. However, efficient LLM inference remains challenging due to the quadratic computational complexity of attention in the prefill stage and the memory-bound nature of attention in the decoding stage. Sparse attention techniques partially reduce complexity in the prefill stage, yet substantial data movement between processors and memory (external I/O) continues to limit GPU efficiency in practice. To address these challenges, we propose PILOT, a Processing-in-Memory (PIM)-based, I/O-aware software-hardware co-design that targets memory-bound attention computations. At the hardware level, PILOT strategically integrates processing units within memory to efficiently support hybrid sparse attention in the prefill stage and both dense and sparse attention in the decoding stage, reducing external I/O overhead. At the software level, a detailed I/O analysis guides tiling and scheduling strategies that optimize buffer utilization and minimize internal I/O within the PIM architecture. Finally, we establish a GPU-PILOT heterogeneous system, delivering an efficient end-to-end solution for long-sequence LLM inference. Evaluation results demonstrate that PILOT achieves up to 19.96× and 57.11× speedups over the GPU baseline for sparse attention in the prefill stage and attention in the decoding stage, respectively. The full GPU-PILOT heterogeneous system further improves end-to-end long-sequence LLM inference by up to 5.31× compared to a conventional GPU system.
Personalized federated learning (PFL) has shown great potential for addressing client heterogeneity in non-independent and identically distributed (non-IID) scenarios. However, achieving effective personalization while preserving privacy remains challenging. Existing PFL with differential privacy (DP-PFL) approaches face two major challenges. First, in DP-PFL, local shared parameters are often directly replaced with those from the global model, but the inconsistency between global and local parameters, together with the noise injected into the global model, can diminish the effectiveness of local models. Second, conventional aggregation strategies typically assign weights solely based on data size. Consequently, models trained on highly personalized clients may introduce biased parameters during aggregation, leading the global model to overfit to client-specific patterns and undermining its generalization ability. To overcome these two challenges, we propose a novel framework, Federated Dynamic Parameter Alignment and Adaptive Aggregation (FedDAAA). Specifically, we partition personalized and shared parameters using the Fisher information matrix. We then perform parameter alignment before replacing shared parameters, which mitigates the adverse effects of noise and parameter discrepancies. Furthermore, we design a new metric that captures client differences by quantifying the proportion of personalized parameters within the entire model. This metric is employed to assign aggregation weights, thereby enhancing the generalization capability of the global model. Extensive experiments and ablation studies demonstrate that FedDAAA significantly outperforms existing baselines across diverse tasks and datasets, effectively enhancing performance while ensuring privacy.
Mixture-of-Experts (MoE) models offer computational efficiency during inference by activating only a subset of specialized experts for a given input. This enables efficient model scaling on multi-GPU systems that use expert parallelism without compromising performance. However, load imbalance among experts and GPUs introduces waiting times, which can significantly increase inference latency. To address this challenge, we propose HarMoEny, a novel solution to address MoE load imbalance through two simple techniques: (i) dynamic token redistribution to underutilized GPUs and (ii) asynchronous prefetching of experts from the system to GPU memory. These techniques achieve a near-perfect load balance among experts and GPUs and mitigate delays caused by overloaded GPUs. We implement HarMoEny and compare its latency and throughput with four MoE baselines using real-world and synthetic datasets. Under heavy load imbalance, HarMoEny increases throughput by 37%–70% and reduces time-to-first-token by 34%–41%, compared to the next-best baseline. Moreover, our ablation study demonstrates that HarMoEny’s scheduling policy reduces the GPU idling time by up to 84% compared to the baseline policies.
The scaling of transformer-based Large Language Models (LLMs) has significantly expanded their context lengths, enabling applications where inputs exceed 100K tokens. Our analysis of a recent Azure LLM inference trace reveals a highly skewed long-tail distribution of input lengths, with approximately 10% of inputs longer than 4K tokens. Existing cluster-level LLM scheduling strategies, including First-In-First-Out (FIFO), reservation-based, and priority-based approaches, primarily target short-input requests with lengths below 4K and fail to address this heterogeneity, leading to inefficiencies such as head-of-line blocking, resource underutilization, and starvation of long-input requests. We propose PEACE, a PreEmptive And efficient Cluster schEduling system for LLM inference, to improve the performance of short-input requests with minimal impact on the Job Completion Time (JCT) of long-input requests. PEACE introduces the following key techniques: 1) preemptive scheduling that prioritizes short-input requests for their performance; 2) coordinated prefill-decode colocation and disaggregation, which reduces both the duration and frequency of preemptions; 3) fast Sequence Parallelism (SP) that minimizes the prefill time of long-input requests to further reduce the likelihood and frequency of preemptions. Evaluations based on Azure LLM inference trace show that, compared to state-of-the-art cluster-level LLM inference schedulers, PEACE reduces the 99th percentile queueing delay of short-input requests by up to 92% and improves their throughput by up to 595%, without significantly affecting the JCT of long-input requests. We open-sourced our code.
Confidential Computing (CC) is becoming a de facto security standard in data centers, as evidenced by the growing number of hardware platforms that offer CC capabilities. CC addresses the needs of applications that process sensitive data across fields such as artificial intelligence (AI), finance, and healthcare. As we observe the convergence of cloud and high-performance computing (HPC), confidential HPC (cHPC) is becoming increasingly crucial for meeting the performance requirements of these compute-intensive workloads. Yet, the current understanding of end-to-end performance and cost trade-offs of cHPC’s hardware remains unclear, despite being essential for the broad adoption and design of cHPC systems. We address this gap by introducing SecPerf, the first performance benchmarking platform tailored explicitly to secure cHPC solutions. Since performance is directly tied to the guarantees provided, SecPerf introduces a taxonomy of security protections that clarifies what is offered at what cost. SecPerf also categorizes the abundant security hardware in modern HPC nodes and supports extensibility by leveraging the Docker framework. We demonstrate SecPerf’s potential by evaluating the performance and cost of primitives, secure networking, offloading, computing, and acceleration on state-of-the-art hardware across 25 algorithms, 18 network primitives, and configuration variants of 29 modern applications. Our results highlight key performance bottlenecks and deepen understanding of the cost of cHPC. By providing a unified benchmarking platform and empirical insights, SecPerf lays the groundwork for the future development and adoption of cHPC.
Federated Learning (FL) operates at scales ranging from thousands to millions of devices, each with heterogeneous data characteristics and computational capabilities. Incorporating unlabeled clients into FL reflects a more realistic distributed learning scenario, as large volumes of data are continuously generated on endpoint devices where manual labeling is impractical and raises privacy concerns. However, most existing approaches assume supervised settings and rely on client-side loss for participant selection, a strategy that becomes unreliable with unlabeled data and often leads to poor convergence. In this paper, we propose MosaicUFL, a framework that leverages cosine similarity as the primary metric for participant selection and replaces traditional loss-based guidance. We further introduce an intra-group synchronous and inter-group asynchronous strategy to improve both statistical and system efficiency. Theoretically, we show that MosaicUFL converges to a bounded neighborhood of the optimal supervised solution under pseudo-label noise and asynchronous updates, which explains the stability of similarity-based filtering. Extensive evaluations demonstrate that MosaicUFL achieves consistent accuracy improvements of up to 10.84% over existing selection mechanisms and effectively meets unsupervised deployment requirements at large scale.
Efficient batch job scheduling is crucial to enhancing the performance of HPC platforms. Existing schedulers, including strategies based on heuristic priority functions and schedulers based on reinforcement learning (RL), face certain limitations. Heuristic approaches lack adaptability to diverse workloads, while RL-based methods suffer from convergence instability and cumbersome training procedures. To address these challenges, we propose an imitation learning-based job scheduler (ILOS) for batch job scheduling, combining the efficiency of learning-based methods with the high solution quality of combinatorial optimization solvers. ILOS utilizes an imitation learning module trained on solver-annotated traces to emulate near-optimal scheduling solutions. During online job scheduling, the emulated solutions serve as initial inputs to our solver to search for higher-quality results within response latency constraints. Extensive evaluations on real-world and synthetic traces demonstrate that ILOS consistently outperforms state-of-the-art heuristic and RL-based schedulers, significantly enhancing HPC cluster efficiency. By isolating the high computational cost into offline annotation and training, ILOS achieves a balance between efficiency and solution quality, making it a practical and adaptable scheduler for HPC platforms.
Scientific codes that process large amounts of data often leverage GPUs for performance gains, but the relatively small global memory capacity can be problematic. As a remedy, we present SLEEK, a software-based main-memory compression technique for single- and double-precision floating-point data that works on existing systems. Its high speed, comparable to that of memory copies on GPUs, makes it a promising solution for scientific applications grappling with memory-size limitations. SLEEK provides both lossless and guaranteed-error-bounded lossy compression that is CPU–GPU compatible. It supports all IEEE 754 floating-point values, including infinities, NaNs, and subnormals. SLEEK’s lossy compressor yields geometric-mean compression ratios of up to 49.8 and 98.9 on single- and double-precision SDRBench inputs. On an RTX 4090 GPU, it compresses up to 1.71× and decompresses up to 1.65× faster than the speed of device-to-device memory copies of the uncompressed data. SLEEK exceeds the throughput of existing GPU compressors while also yielding higher compression ratios than many of them.
Heterogeneous graph neural networks (HGNNs) have exhibited remarkable performance across diverse graph-based applications. However, training HGNNs on massive heterogeneous graphs encounters critical CPU-GPU communication bottlenecks. Specifically, existing systems suffer from: (1) substantial CPU-GPU read overhead resulting from frequent transfers of high-dimensional node features, and (2) prohibitive GPU-CPU write transfer costs caused by updating learnable embeddings for featureless nodes. Caching is commonly adopted to alleviate these bottlenecks. Nevertheless, existing cache-based methods are severely limited by low cache ratios at large data scales and suboptimal designs that fail to address the heterogeneous access patterns inherent in HGNN training.To overcome these limitations, we propose MeCache, a system that mitigates CPU-GPU communication overhead in multi-GPU HGNN training. MeCache introduces two key innovations: (1) a semantic-aware feature reduction strategy that employs type-specific dimensional compression to minimize feature transfer while preserving model accuracy, and (2) a fused caching method that implements differentiated cache structures tailored to distinct data access patterns, along with a cost model for efficient cross-GPU cache allocation. Evaluations on two representative HGNN models and five datasets show that MeCache reduces CPU-GPU communication overhead by 84% on average and achieves an average 2.7× speedup compared to state-of-the-art systems (DGL and Heta), with negligible accuracy loss.
Serving large language models (LLMs) is highly resource-intensive and requires specialized hardware acceleration. While GPUs remain the dominant solution, dataflow accelerators are emerging as a compelling alternative. In this study, we present an extensive empirical performance analysis of six datacenter-class GPUs from Nvidia, AMD, and Intel, alongside two dataflow AI accelerators from Cerebras and SambaNova, using fourteen different open-source LLMs. Our evaluation focuses on investigating the main factors that impact LLM inference, including model size, batch size, quantization, and multi-GPU scaling under various parallelism strategies. Importantly, we analyze both performance and energy efficiency, providing an energy-aware comparison across accelerator types. Our experimental results show that dataflow AI accelerators achieve an order-of-magnitude improvement in throughput and latency for small (≤ 8) batch sizes compared to GPUs. On the other hand, GPUs offer larger HBM memories and benefit from a simpler programming model, allowing for more flexibility in terms of batch size, which is particularly beneficial in offline inference. These insights provide practical guidance for LLM inference in diverse deployment scenarios.
Parallelizing sparse matrix computations with loop-carried dependencies, such as sparse triangular solve and incomplete factorizations, is critical for scientific computing, yet remains challenging on modern multi-core architectures due to irregular data dependencies. These dependencies complicate task scheduling, making it difficult to maximize data locality, ensure load balance, and minimize synchronization overhead. This paper presents BAAS, a novel framework for shared-memory systems that generates high-performance parallel schedule by addressing these trade-offs. BAAS features two core contributions. First, we propose a Bidirectional Subtree Aggregation scheme that captures both divergent and convergent node patterns to improve data locality and reduce synchronization by grouping strongly dependent nodes. Second, we introduce an Affinity-Aware Scheduling strategy that uses adaptive wavefront coarsening to allocate an appropriate number of threads while merging consecutive parallel bottlenecks, and employs node-thread affinity-aware mapping to assign nodes to threads, minimizing inter-thread synchronizations and maintaining load balance. The experimental results show that BAAS consistently outperforms vendor-libraries and state-of-the-art methods in SpTRSV, SpIC0, and SpILU0, achieving superior efficiency. Furthermore, when integrated into the ICCG iterative solver, BAAS further proves its ability to significantly enhance the end-to-end solver performance.
Computing infrastructure is increasingly composed of non-uniform memory access (NUMA) architectures, such as machines with multiple sockets and rackscale systems with shared memory. A powerful technique to effectively use such architectures is to replicate data structures across NUMA domains, so threads can quickly access a local replica of data. Unfortunately, existing replication methods support only a static set of replicas, which is mismatched to the dynamic needs of modern systems. We introduce LD-NR, a replication technique that allows the number and location of replicas to change dynamically. LD-NR can adapt to workloads that change over time, such as applications that scale up and down across NUMA domains, whereas NR, the state of the art in NUMA-aware data structure replication, is limited to a static replica set. LD-NR achieves this benefit with negligible memory and performance overhead when configured identically to NR. We show that a hashmap dynamically replicated with LD-NR achieves 2.6 times better throughput than the best performing concurrent hashmap implementation, and provides 97% of the throughput provided by NR with an ideal, static configuration. We further demonstrate the utility of LD-NR to optimize complex systems by using it to enable a granular, per-process page table replication policy in a rackscale operating system.
As large language models (LLMs) continue to scale and are increasingly applied to long-context inference tasks, the memory consumption of the key-value (KV) cache has become a critical bottleneck on GPUs. Quantization offers a promising solution, yet existing methods often suffer from high memory overhead, inefficient GPU implementations, and expensive offline preprocessing. In this paper, we introduce Vayne, a unified GPU kernel that seamlessly integrates quantization and lossless compression, enabling near-zero-cost KV cache reduction. Vayne also features a highly optimized lossless GPU encoding scheme that substantially reduces memory usage while preserving model accuracy. At the same time, our implementation incorporates both kernel-level and algorithm-level optimizations, including bit transposing, a decoupled lookback prefix-sum mechanism, pre-RoPE strategies, and other computational optimizations. Experimental results on LongBench using the LLaMA 3.1 8B Instruct model demonstrate that Vayne achieves an average 8.4× speedup over the next best baseline, a 1.4× lower bit rate, and higher accuracy under comparable bit rates. These results highlight Vayne’s effectiveness in enabling high-throughput, memory-efficient LLM inference at scale through efficient KV cache compression.
Graph Neural Networks (GNNs) have been widely adopted for their ability to compute expressive node representations in graph datasets. However, serving GNNs on large graphs is challenging due to the high communication, computation, and memory overheads of constructing and executing computation graphs, which represent information flow across large neighborhoods. Existing approximation techniques in training can mitigate the overheads but, in serving, still lead to high latency and/or accuracy loss. To this end, we propose Reforge, a system that enables low-latency GNN serving for large graphs with minimal accuracy loss through two key ideas. First, Reforge employs selective recomputation of precomputed embeddings, which allows for reusing precomputed computation subgraphs while selectively recomputing a small fraction to minimize accuracy loss. Second, we develop computation graph parallelism, which reduces communication overhead by parallelizing the creation and execution of computation graphs across machines. Our evaluation with large graph datasets and GNN models shows that Reforge significantly outperforms state-of-the-art techniques.
