As the field of quantum computing moves towards the realization of Quantum Error Correction (QEC), an increasing amount of attention has been paid to the implementation of decoders-classical systems analyzing the state of error on the quantum device in real-time. However, today there is a lack of tools available to characterize and compare the performance of decoders. In this work, we address this need and introduce decoder-bench, a framework for benchmarking decoders on relevant QEC code traces. decoder-bench integrates with Stim, a high performance tool for analyzing stabilizer circuits, to create traces for a variety of QEC codes and fault-tolerant subroutines. We use decoder-bench to evaluate the accuracy and latency performance of multiple decoders for circuit-level simulations of color code memory experiments, bivariate-bicycle code memory experiments, surface code memory experiments, and surface code lattice surgery experiments.
The increasing complexity and performance requirements of extended reality (XR) platforms necessitate a full-stack approach to optimize system architectures. We present XRSight, a hardware-software co-design framework for developing and characterizing extended reality (XR) workloads on embedded systems-on-chip (SoCs). This open-source tool integrates a cycle-accurate, FPGA-accelerated simulation environment with a representative XR application suite to enable architectural exploration and detailed performance analysis. By bridging real-world XR workloads with a modular system-on-chip (SoC) platform, our framework offers fine-grained visibility into SoC execution behavior across compute, memory, and interconnect subsystems. We demonstrate the utility of our work through case studies that examine how different hardware configurations and workload mappings affect end-to-end system performance. Our results show how the tool can inform hardware design decisions and workload optimizations. This framework lowers the barrier to entry for researchers and architects exploring XR-centric system designs in an open and extensible SoC ecosystem.
Cheaper and more accurate sequencing technologies have led to a large volume of genetic data that poses significant computational challenges and requires novel computing solutions to keep pace. This increased volume has also enabled the use of pangenome graph references, which provide better quality alignments because they represent variation, but they require new algorithms that are usually slower than those using a traditional reference genome, and exhibit different computational characteristics.We introduce PangenomicsBench, the first benchmark suite targeting computational pangenomics, with six CPU and two GPU kernels extracted from popular tools, designed to guide future research in pangenomics software and hardware acceleration. We characterize these workloads to reveal the following key insights: (a) Seq2Graph mapping algorithms are limited by control complexity rather than memory access to the reference graph because they process small, cache-friendly subgraphs. (b) GPUs have the potential for large speedups, but are limited by control divergence for mapping workloads. (c) Pangenomics introduces computational patterns different from traditional genomics like stochastic gradient descent. (d) Pangenomic mapping algorithms are highly sensitive to reference graph structures. (e) There are opportunities for optimizing existing software.
Serverless computing has emerged as a competitive cloud computing paradigm. At the same time, the open-source RISC-V ISA has gained a lot of interest and the first RISC-V systems have already started to appear in the server market for datacenters. The combination of these computing trends necessitates the performance assessment of the impact of the RISC-V ISA and relevant processor implementations when executing serverless workloads. However, currently there is no benchmarking support for systematically evaluating serverless workloads on RISC-V systems. In this paper we bridge this gap in benchmarking support across the layers of the computing stack, from the application to the microarchitecture. We port the vSwarm serverless benchmark suite to the RISC-V ISA and enable the execution of the workloads in both real and simulated RISC-V platforms using the gem5 microarchitectural simulator. To demonstrate the usefulness of the infrastructure, we quantify the performance trade-off of cold vs. warm execution on a simulated RISC-V system and validate the results against a real platform. Overall, our enhanced benchmarking support creates new opportunities for further experimentation with serverless workloads on RISC-V systems, enabling the analysis and optimization of their performance across the computing stack.
Modern storage workloads commonly run in containers within data centers, such as machine learning, databases, caches, HPC, and serverless workloads. To facilitate the storage performance requirements (e.g., bandwidth, latency) of these workloads, data centers have adopted fast NVMe SSDs as a storage medium. At the same time, data centers virtualize and share these storage resources with multiple tenants to improve resource utilization and reduce costs. Such sharing leads to an inherent trade-off between tenant performance isolation and SSD utilization. Although various research studies demonstrate how to achieve various performance isolation properties, such as fairness, there is neither a unified definition for performance isolation nor a benchmark. Furthermore, the isolation capabilities of state-of-the-practice I/O control mechanisms in the Linux kernel are not well understood. In this paper, we address these three challenges. First, we survey the definition of performance isolation and uncover four common performance isolation desiderata. Second, we introduceisol-bench, a benchmark for evaluating these desiderata for I/O control mechanisms. Third, we use isol-bench to evaluate I/O isolation for Linux’s state-of-the-practice I/O control mechanism, cgroups. From our evaluation, we are able to conclude that out of cgroups’s knobs io.cost achieves the most isolation desiderata, but has a latency overhead past CPU saturation. We open-source the source code of isol-bench at https://github.com/atlarge-research/isol-bench.
Blockchains have revolutionized trust and transparency in distributed systems, yet their heavy reliance on key-value (KV) storage for managing immutable, rapidly growing data leads to performance bottlenecks due to I/O inefficiencies. In this paper, we analyze Ethereum’s storage workload traces, with billions of KV operations, across four dimensions: storage overhead, KV operation distributions, read correlations, and update correlations. Our study reveals 11 key findings and provides suggestions on the design and optimization of blockchain storage.
EntoBench is the first open, MCU-ready benchmark suite and evaluation framework that captures the full insect-scale robot pipeline. Thirty-one kernels, each configurable for float, double, or fixed-point arithmetic, map how milliwatt power budgets and tight memory constraints can reshape algorithmic trade-offs. A synchronized GPIO harness couples logic-analyzer timing with inline current sensing, allowing any Cortex-M0+, M4, M33, or M7 board to report latency, energy, and peak power under cache-on/off condition and varied kernel parameters. More than 400 experiment runs reveal systematic patterns linking architecture features to achievable autonomy, providing a rigorous baseline for future software optimization and hardware-software co-design in this emerging cyber-physical domain. The full framework and benchmark suite are released as open source.
Workload characterization plays a crucial role in analyzing applications, understanding their behavior, building benchmark suites, and evaluating performance across various architectures. Existing benchmark suites are frequently capped in size due to the time-consuming creation procedure, which limits the number of samples in a dataset. Profiling multiple characteristics for each benchmark in order to better characterize and understand its behavior results in a high feature-to-sample ratio. Traditional workload characterization techniques, such as those employing principal component analysis (PCA), can be challenging in high-dimensional spaces, particularly when dealing with limited sample sizes. This paper presents a novel multilevel PCA approach that addresses these issues. Our approach extracts information from three unique layers: microarchitecture-independent, compiler intermediate representation (IR), and source code layers, to provide a more comprehensive and accurate representation of application characteristics.Our methodology, cross-layer multilevel PCA (CMPCA), applies PCA hierarchically at multiple levels to effectively reduce dimensionality while preserving crucial information. Furthermore, we integrate clustering techniques to identify representative benchmarks, significantly reducing the dataset size (53.8X). The results demonstrate that our approach outperforms standard PCA and single-layer characterization approaches, achieving higher accuracy in speedup calculations for representative benchmarks compared to the entire dataset. This work provides a reliable and efficient method for reducing datasets and characterizing workloads across a variety of architectures.
Keeping up with the soaring memory demands of Large Language Models (LLMs) is one of the biggest hurdles in modern system design. Given their size, distributed training across multiple GPUs is no longer optional but required. As distributed training takes center stage, unlocking parallelism across devices is more important than ever. Still, even the best parallelism techniques face roadblocks imposed by limited memory capacity. Unified Virtual Memory (UVM) o˛ers a compelling solution by automating data transfers between host and device via demand paging, thereby supporting memory oversubscription. Despite its potential, the integration of UVM with various parallelism strategies in distributed training has yet to be thoroughly investigated. This study explores new opportunities for leveraging UVM in distributed LLM training. Our analysis shows that finer-grained parallelism accelerates training performance under UVM, although the increased communication between small partitions demands careful attention. This approach also yields better memory efficiency with fewer page faults and contributes to improved scalability. As the first in-depth analysis of parallelism in distributed training with UVM, this work provides key insights into how UVM can be effectively integrated alongside diverse parallelism strategies for training large-scale LLMs.
The exponential growth of large language models has outpaced the capabilities of traditional CPU and GPU architectures due to the slowdown of Moore's Law. Dataflow AI accelerators present a promising alternative; however, there remains a lack of in-depth performance analysis and standardized benchmarking methodologies for LLM training. We introduce DABench-LLM, the first benchmarking framework designed for evaluating LLM workloads on dataflow-based accelerators. By combining intra-chip performance profiling and inter-chip scalability analysis, DABench-LLM enables comprehensive evaluation across key metrics such as resource allocation, load balance, and resource efficiency. The framework helps researchers rapidly gain insights into underlying hardware and system behaviors, and provides guidance for performance optimizations. We validate DABench-LLM on three commodity dataflow accelerators, Cerebras WSE-2, SambaNova RDU, and Graphcore IPU. Our framework reveals performance bottlenecks and provides specific optimization strategies, demonstrating its generality and effectiveness across a diverse range of dataflow-based AI hardware platforms.
Data processing on the edge offers advantages over cloud-based solutions, including reduced latency and lower energy consumption. However, to fully utilize edge SoCs, applications must be efficiently mapped onto these devices’ constituent heterogeneous processing units. This mapping is challenging, as modern SoCs incorporate diverse compute units, such as big.LITTLE architectures and GPUs with distinct performance characteristics. Furthermore, due to edge SoCs’ integration and resource constraints, execution on one processing unit can interfere with the runtime of others, complicating the construction of modular and composable performance models.To address these challenges, we present BetterTogether, a flexible scheduling framework that enables fine-grained software pipelining on heterogeneous SoCs. Applications are provided as a sequence of stages, each with a CPU and GPU implementation. These stages can then be pipelined across the various processing units on the SoC. The novel component of BetterTogether is its ability to generate accurate and efficient pipeline schedules using a profile-guided performance model that captures execution time under representative intra-application interference. We demonstrate the portability of BetterTogether by evaluating it on three SoCs with GPUs from different vendors (NVIDIA, Arm, and Qualcomm) and using three computer vision edge workloads with different computational characteristics. Our performance model yields predictions that correlate strongly with measured results. Using these models, we construct efficient pipeline schedules specialized to each workload-platform combination, outperforming homogeneous GPU baselines in nearly all cases, with a geomean speedup of 2.14x and a maximum of 7.59x. Finally, we present a preliminary exploration with Google’s EdgeTPU, highlighting BetterTogether’s extensibility to incorporate an even wider variety of processing units into its backend.
Large, real-world scientific applications are often complex, making them difficult to analyze, characterize, and optimize. Such applications typically involve intricate I/O patterns and library dependencies, which can make workflow analysis and tuning difficult. Proxy applications offer a practical solution by emulating the essential characteristics of the original application while significantly reducing complexity.In this work, we present miniGiraffe, a proxy application for Giraffe, a sophisticated genomics tool that operates over a pangenome, a graph-based structure capturing genetic variation across a species. We develop miniGiraffe using a principled methodology: carefully characterizing Giraffe’s behavior and validating that our proxy faithfully reproduces its key computational features. miniGiraffe contains only 2% of Giraffe’s codebase, while producing identical outputs for the most computationally intensive code components and closely matches Giraffe’s execution time and scaling behavior in these regions. The simplified design of miniGiraffe enabled rapid experimentation across multiple architectures, which we utilize to perform an autotuning experiment of the mapping workflow; we found that specializing parameters to inputs and architectures provided a geometric mean speedup of 1.15× and up to a 3.32× speedup over the default parameters.
Comparing the security of cache side-channel defenses is challenging. These defenses are often evaluated in performance-oriented simulators, which lack the ability to also measure security. Recent security quantification methods are typically implemented in custom frameworks that do not report performance. Furthermore, prior work either lacks in generalizability or adaptability, making it difficult for computer architects to evaluate the robustness of the design or for non-security experts to reason about the security challenges. To address these gaps, we propose a generalizable and accessible cache security metric, the Cache Access Security Metric (CASM). CASM measures how much leakage is present in a given cache architecture using a variety of characteristics. We then incorporate CASM into the popular processor simulator, gem5. We evaluate the security of various cache architectures, secure and non-secure, using CASM to demonstrate its potential.
The rapid growth of mobile gaming and streaming has intensified the demand for energy-efficient smartphones. Modern gaming and streaming applications rely on complex 3D rendering to provide an immersive user experience, which is a compute-intensive process. To address this energy challenge, this work analyzes the microarchitectural behavior of 3D rendering for a mobile game and explores adaptive graphical settings as a new approach to improving energy efficiency. We collect the energy, performance, and temperature of the System-on-Chip (SoC) and memory, along with microarchitectural data, from a Google Pixel 8a smartphone running the "Boat Attack" benchmark to evaluate and understand its behavior. Subsequently, we analyze the results and devise a static graphics scaling approach to adapt to the application behavior. Experimental results show a 43% reduction in energy consumption of the benchmark, while maintaining image similarity scores comparable to high settings.
Retrieval-Augmented Generation (RAG) has emerged as a popular technique for addressing several challenges of Large Language Model (LLM) systems, including static model knowledge, hallucination, and limited input sequence lengths. Although RAG mitigates common pitfalls of current LLM systems, its inherent heterogeneity and configurability introduce new challenges. The performance of RAG is crucial for meeting the high-throughput and low-latency demands of LLM services. Different components of RAG operate on different hardware platforms, and their complexity scales with the configurability and complexity of the rest of the system. For example, larger embeddings may enhance retrieval accuracy, but also increase the latency of embedding creation and indexing, thereby compromising the RAG system’s performance and energy consumption.Thus, a comprehensive characterization of an end-to-end RAG system becomes necessary. In this work, we build an end-to-end RAG benchmarking framework, Athena, that supports various embedding models, vector databases, index/search algorithms, and LLMs. By characterizing the system under various RAG settings built using Athena, we demystify RAG by identifying performance bottlenecks and quantifying the impact of each sub-component on overall system performance. In addition, the plug-and-play, open-sourced Athena framework is designed to assist future RAG research.
Modern Graphics Processing Units (GPUs), such as NVIDIA’s Hopper and Blackwell, leverage Thread Block Clusters (TBCs) to enhance performance and resource management. TBCs introduce a hierarchical organization, grouping thread blocks into clusters that enable efficient synchronization and distributed shared memory access. This innovation improves data locality and reduces latency in inter-thread block communication, unlocking new opportunities for executing complex parallel workloads. However, modeling the intricate interactions within TBCs, especially the balance between data locality and resource contention, is challenging. This is further complicated by limited access to cutting-edge hardware like Hopper GPUs, which restricts direct experimentation. As a result, robust simulation models are needed to accurately replicate TBC behavior. This paper presents a detailed simulation model that captures TBC performance characteristics. Our model enables researchers to explore TBC functionalities and evaluate performance implications without requiring physical Hopper GPUs. Validation against an NVIDIA H100 GPU shows a Mean Absolute Relative Error (MARE) of 4.7%, demonstrating the model’s accuracy and utility for advancing research in GPU architectures and parallel computing.
The memory footprint of modern applications like large language models (LLMs) far exceeds the memory capacity of accelerators they run on and often spills over to host memory. As model sizes continue to grow, DRAM-based memory is no longer sufficient to contain these models, resulting in further spill-over to storage and necessitating the use of technologies like Intel Optane and CXL-enabled memory expansion. While such technologies provide more capacity, their higher latency and lower bandwidth has given rise to heterogeneous memory configurations that attempt to strike a balance between capacity and performance. This paper evaluates the impact of such memory configurations on a GPU running out-of-core LLMs. Starting with basic host/device bandwidth measurements using an Optane and Nvidia A100 equipped NUMA system, we present a comprehensive performance analysis of serving OPT-30B and OPT-175B models using FlexGen, a state-of-the-art serving framework.Our characterization shows that FlexGen’s weight placement algorithm is a key bottleneck limiting performance. Based on this observation, we evaluate two alternate weight placement strategies, one each optimizing for inference latency and throughput. When combined with model quantization, our strategies improve latency and throughput by 27% and 5x, respectively. These figures are within 9% and 6% of an all-DRAM system, demonstrating how careful data placement can effectively enable the substitution of DRAM with high-capacity but slower memory, improving overall system energy efficiency.
The rapidly expanding volume of genomic datasets causes processing speed to become a significant bottleneck in genomic analysis workflows. FPGA acceleration presents an effective means to optimize computationally intensive workloads. K-mer counting is a widely used operation that involves recording each substring within a nucleotide sequence. This is frequently done with a counting Bloom filter, a data structure that requires multiple hash computations and memory accesses to a large lookup table per K-mer operation. This work presents two primary contributions. First, we profile Bloom filter implementations, examining hash rate performance, memory access patterns with and without prefetching, and the impacts of sorting to identify scaling bottlenecks and optimal resource allocation strategies. Second, we present CoBloom, an FPGA-accelerated counting Bloom filter which employs a hybrid design combining hardware-based hash computation with CPU-managed memory operations. CoBloom’s architecture addresses the identified bottlenecks to create a more efficient K-mer counting pipeline.
In recent years, graph neural networks (GNNs) have emerged as powerful tools for analyzing graph-structured data. However, as graph sizes grow, GNNs face significant scalability challenges. To address this problem, previous work has proposed sampling and sparsification techniques aimed at enabling training on subsets of the original graph rather than on the entire graph. These mechanisms introduce trade-offs between memory requirements, accuracy, and training time. Moreover, the selection of a suitable sampling or sparsification strategy, as well as its optimal configuration, depends on multiple factors, including the graph size and the GNN model in use. However, a comprehensive and unbiased comparison of different strategies is lacking. Furthermore, these mechanisms have mostly been evaluated on small and medium-sized graphs consisting of a few million vertices or fewer, and not on large-scale graphs that would benefit from themTo fill this gap, this paper presents an extensive evaluation of sampling and sparsification techniques on graph datasets of varying scales from two popular benchmark suites. The study explores the trade-offs mentioned above. Our experiments reveal that selecting an appropriate set of optimizations substantially reduces computational load and memory consumption, resulting in faster training times while maintaining final accuracy. Additionally, we find that stochastic sampling methods reduce convergence time by approximately 55% compared to partition-based techniques. Graph Attention Networks improve final accuracy by about 1% over other models but require up to 3 times more GPU memory. Furthermore, graph sparsification consistently reduces GPU memory usage by over 18%.
