System software architectures, such as assemblers, require synchronous maintenance as Instruction Set Architectures (ISAs) continuously evolve. These software systems typically feature dedicated, specialized implementations for each ISA. However, the fragmentation of data structures and implementation styles across distinct ISAs complicate performance optimization, rendering these optimizations non-portable, and introduce systemic challenges for rigorous performance analysis. Analogous issues also plague software like compiler backends.To address this challenge, we propose the Scalable Assembler System (SAS). SAS enables the reuse of ISA-agnostic modules by unifying its core data structures and procedural definitions. The SAS architecture, through the standardization of key data structures like instruction descriptors and unified workflow, decouples the core stages of the assembly process from ISAspecific designs. This separation allows performance analysis and optimization methodologies to be systematically applied across the entire assembly architecture.We have implemented a prototype system in the Rust language supporting AMD64, AArch64, and RISC-V64. Performance analysis to this prototype initially indicates that it exhibits a statistically significant performance advantage over GNU AS. Our quantitative analysis shows that integrating SAS into the compilation pipeline can yield approximately $1 \%$ performance improvement. We believe that the design experience of SAS and its unified methodology for performance analysis hold promise for further generalization to the architecture and optimization of other system software.
High-performance parallel acceleration necessitates high-performance CPUs to stave off Amdahl’s law. Getting the most out of a CPU requires instruction-level performance profiling to understand microarchitecture performance bottlenecks, guiding hardware design for future generations. This depends on effective methods for using instruction profilers to identify the key ways performance losses occur in out-of-order processors. Identifying the root causes of these losses is challenging, but can be greatly assisted by comprehending how they manifest behaviorally in hardware, which we refer to in this work as ‘performance pathologies’. Historically, characterizing these behaviors has been a labor-intensive process that required a detailed understanding of software and microarchitecture. However, the advent of profilers capable of producing accurate instruction-level cycle stacks at both ends of the execution window, i.e., at dispatch and commit, dramatically simplifies this task. In this work, we present Pesto, an instruction-level methodology for diagnosing microarchitecture performance pathologies, i.e., identifying the key microarchitectural manifestations of performance loss in out-of-order processors. Pesto uses k-means clustering to group per-instruction cycle stack components produced at both ends of the execution window. The resulting centroids for each cluster characterize exactly one kind of pathology (e.g., pipeline flush due to branch misprediction, or commit stall due to data cache misses). We apply Pesto to an FPGA-accelerated BOOM core in FireSim running SPEC2017 and discover that the BOOM core has 20 pathologies, corresponding to 20 fundamental ways its microarchitecture can lose performance. Furthermore, we observe that individual instruction types (e.g., loads) lead to performance loss in at most four ways. Finally, we show that Pesto works using statistical sampling, making it easy to adopt.
As GPUs have grown more popular and become commonplace in datacenters, GPU power consumption has become a more important metric than ever before. Power behavior should be a major consideration in all stages of GPU design, development, and deployment. However, accurate power evaluation is difficult because modern-day GPU workloads are large and complicated, and rapid advances in popular datacenter workloads means that the power profiles of these workloads change often. To solve this problem, we introduce Power Ranger, a GPU power benchmark generator that can replicate power behavior from any arbitrary workload using only a reference device and a trace of power over time. Power Ranger’s generated workloads replicate real workload behavior using simple kernel launches selected from a look-up table. Because Power Ranger only requires a power trace and device, and does not require any code from the target workload, it can also create real GPU workloads that match any desired power behavior. This capability enables the generation of custom power benchmarks and stress tests that can be executed on real GPU hardware. We evaluate Power Ranger using several real and synthetic workloads, and we find that Power Ranger is able to reproduce original power traces with a MAPE of $5 \%$ or less for all cases.
The end of Dennard scaling and the imminent end of Moore’s law is making it increasingly critical to write software that fully utilizes the hardware compute resources. Software developers hence need to gain insight into the inefficiencies of their applications, and a typical first step is to obtain applicationlevel Cycles-Per-Instruction (CPI) stacks, for example by using a Top-Down methodology. Unfortunately, existing approaches have only been validated against processors with idealized microarchitectural resources. Although this approach adds confidence that the generated CPI stacks make intuitive sense, it does not conclusively document that the CPI stacks break down total execution time into components in an architecturally sound manner. Our goal in this paper is to bridge this knowledge gap. We first define a complete and mutually exclusive reference that accounts for all dispatch slots in all clock cycles across all microoperations ($\mu$ ops), i.e., a $\mu$ op is either (i) dispatching, (ii) stalled while waiting for a structural back-end stall, (iii) not available due to a front-end issue, or (iv) mistakenly executed due to misspeculation. If the reference is also architecturally sound, i.e., it attributes each dispatch slot to the CPI stack component that accurately represents the impact on effective dispatch bandwidth, we refer to it as foundational; second-order overlap effects typically mean that an architecture has multiple foundational references. We derive foundational references for the BOOM out-of-order core, and, surprisingly perhaps, find that they can be implemented in hardware with less than 1 KB of additional state. We hence propose the Foundational Yet Implementable (FYI) counter architecture for CPI stacks that are, by design, an exact match for the target reference.
Architectural simulators are indispensable for modern computer architecture research, but they remain notoriously slow due to their event-driven, cycle-level execution model. In this work, we present a set of software- and system-level optimizations to accelerate large-scale design-space exploration with gem5. First, we reduce per-instance simulation time via compiler-level optimization. We demonstrate that although gem5 suffers severe frontend stalls on modern CPUs stemming from its large instruction footprints, naïve Profile-Guided Optimization (PGO) is impractical in this setting because it requires frequent reprofiling and recompilation. To address this, we challenge the conventional reliance on self-profiling and instead construct a universal, performance-driven profile that generalizes across simulation inputs. Second, we improve aggregate simulation throughput by strengthening performance isolation using Sub-NUMA clustering (SNC). Finally, we show that a simple co-scheduling heuristic has great potential for reducing resource stranding and boosting multi-instance efficiency. Together, these techniques improve single simulation speed by 17 % and aggregate throughput by $27 \%$, making large-scale design-space exploration more practical and efficient.
Reinforcement Learning (RL) learns optimal decision-making policies from experiential (transition) datasets and maximizes the RL agent’s cumulative rewards. In order to improve the RL training efficiency, prior works have studied how to selectively sample certain critical transitions that ultimately lead to better policies and rewards. However, RL workloads still face significant challenges from a systems perspective, particularly when the agent iteratively accesses batches of data from transition datasets, whose growing sizes continue to challenge the memory hierarchy. This results in frequent and costly memory transfers between caches and Dynamic Random Access Memory (DRAM), which negatively impacts the overall training time.In this paper, we propose RACER, our novel caching layerbased optimization for RL training workloads. Firstly, recognizing that the RL agent repeatedly accesses large transition batches from growing datasets, we design a storage-cache that prioritizes critical transitions to fit within the hardware cache hierarchies. This design reduces the memory access times by sampling from a subset of critical transitions and minimizes the costly memory trips to DRAM. Second, we demonstrate how to smartly leverage key metrics (viz., temporal difference error and advantage weighting) to quantify the importance/relevance of transitions during policy optimization and to identify the transition data that would need to be spilled out of (or filled into) the caching system. We also introduce dynamic optimizations to our caching system that minimize the prospect of discarding the critical transitions. Our performance evaluation across three state-of-the-art RL algorithms, under various task environments, and on three different systems, demonstrates that RACER achieves significant optimization time improvements to the RL transition data sampling phase (a speedup of $6 \times$) and end-to-end training time (up to $2 \times$) with comparable rewards.
Heterogeneous processors that combine CPUs, GPUs, and NPUs have become an important response to dark silicon and rising workload demands. The high performance per watt potential of these platforms offers a path to lower the energy consumption of ML applications, which is critical for the sustainable growth of these workloads. However, realizing these benefits requires understanding how different phases of an application interact with the compute and memory behavior of each hardware component. In this work, we use a machine-learning-based speech neuroprosthesis application to demonstrate how heterogeneous hardware can be leveraged to improve end-to-end performance. We introduce techniques for reducing memory usage, scheduling data movement, and managing synchronization, which enable different stages of the application to execute efficiently across the NPU, GPU, and multi-threaded CPU. Our evaluation shows a clear divergence in hardware preferences when optimizing for latency versus throughput per watt, and highlights the role of interference when deploying the application in a pipeline-parallel manner. This work shows that understanding both the application structure and the underlying hardware is essential for effectively exploiting heterogeneity and offers guidance for designing sustainable heterogeneous application mappings.
Cycle-level GPU simulators are valuable educational tools, but existing frameworks are either too complex for students to navigate or too abstract to convey microarchitectural details. We present Macsim Mini, a lightweight cycle-level GPU simulator designed for computer architecture education. By concentrating on the memory hierarchy and thread scheduling rather than detailed compute pipelines, Macsim Mini captures the architectural trade-offs most central to GPU performance in a codebase small enough for students to read and modify within course assignments. Macsim Mini has been deployed in a graduate-level GPU architecture and programming course for seven semesters, serving $\sim 1,000$ students with high completion rates and average scores above 90%.
To mitigate well-studied memory vulnerabilities [14], [21], [24], [25], memory devices may implement secure memory [9], [10], [27], [29], [34], an extension to the memory controller logic that guarantees the confidentiality and integrity of data stored in main memory. The memory encryption engine (MEE) is responsible for decrypting and integrity verifying data that comes from the off-chip memory device [10]. To protect outgoing data, the MEE encrypts and protects the integrity of data in 64B-block granularity. To provide confidentiality, the MEE uses counter-mode encryption (CME) [22], [23], [34]. To verify data’s integrity, the MEE maintains a per-block hashed message authentication code (HMAC) [7], [16] in memory. Additionally, Bonsai Merkle Trees (BMT) are used [27] to protect the encryption counter from replay attacks. The root of the BMT is stored on-chip in trusted hardware and serves as a root of trust for authentications. When fetching data from memory, the requisite metadata (i.e., encryption counter, HMAC, and path through the integrity tree) are also fetched to decrypt the data and authenticate its state against the HMAC and trusted root. This ensures that the processor does not perform computation on any corrupted data and that data is private while off-chip.
Emerging applications such as AR are driving demands for machine intelligence capable of processing continuous and/or long-context inputs on local devices. However, currently dominant models based on Transformer architecture suffers from the quadratic computational and memory overhead, which hinders applications required to process long contexts. This has spurred a paradigm shift towards new architectures like State Space Models (SSMs) and SSM-Transformer hybrid models, which provide near-linear scaling. The near-linear scaling enabled efficient handling of millions of tokens while delivering high performance in recent studies. Although such works present promising results, their workload characteristics in terms of computational performance and hardware resource requirements are not yet thoroughly explored, which limits our understanding of their implications to the system level optimizations. To address this gap, we present a comprehensive, comparative benchmarking of carefully selected Transformers, SSMs, and hybrid models specifically for long-context inference on consumer and embedded GPUs. Our analysis shows that SSMs are well-suited for on-device AI on consumer and embedded GPUs for long context inferences. While Transformers are up to $1.9 \times$ faster at short sequences ($ \lt 8 \mathrm{~K}$ tokens), SSMs demonstrate a dramatic performance inversion, becoming up to $4 \times$ faster at very long contexts ($\sim 57 \mathrm{~K}$ tokens), thanks to their linear computational complexity and $\boldsymbol{\sim} \mathbf{6 4 \%}$ reduced memory footprint. Our operator-level analysis reveals that custom SSM kernels like selective scan despite being hardware-aware to minimize memory IO, dominate the inference runtime on edge platforms, accounting for over $55 \%$ of latency due to their sequential, element-wise nature. To foster further research, we are sharing CPU/GPU profiling traces and have made our characterization framework SSM-Scope open-sourced at https://github.com/sapmitra/ssm-scope
The large branch footprints of contemporary applications easily overwhelm the capacity of Branch target buffers (BTBs). Therefore, to avoid frequent BTB misses and their associated performance penalties, commercial processors feature massive BTBs that require hundreds of KBs to multi-MB storage budgets. Furthermore, storage requirements are increasing at an alarming rate, a trend that is certainly not sustainable. As the bulk of BTB storage budget goes towards storing branch targets, researchers have recently proposed storageefficient schemes for target representation. The state-of-the-art schemes are so effective that branch targets no longer dominate the BTB storage requirements, rather, the tags do. However, there has not been any study on understanding the implications of tag size on performance, storage, and aliasing. This work bridges this gap by performing a comprehensive study of how and why tag size requirements vary for performance- and storage-efficiencyoriented BTB designs across different BTB capacities. The results of the study help BTB designers understand where in BTB to invest any additional storage budget that becomes available in next processor generations. The key findings of this study include: 1) moderately sized BTBs ($\mathbf{2 K}$ to $\mathbf{8 K}$ entries) require larger tags than small ($\mathbf{2 5 6}$ entry) and large (32 K entry) BTBs, 2) increasing the number of ways in the BTB also requires larger tags to fully realize the benefits of the additional ways, and 3) a storage-efficient BTB design reduces tag storage requirements by up to $21 \%$ over a performance-oriented BTB design.
In chiplet-based processors, integrating 3D V-Cache into a subset of core complex dies (CCDs) naturally creates cache asymmetry across them. Such asymmetry can lead to nonintuitive performance behavior across different CPU topology configurations, depending on the workload. This work analyzes the performance implications of cache-asymmetric processors by evaluating the AMD 3D V-Cache processor under various topologies. Our evaluation results first show that simply using more cores or threads can sometimes degrade performance due to contention in shared resources. In addition, leveraging a CCD equipped with large V-Cache can underperform compared to a standard CCD with a smaller cache capacity because of its lower clock frequency. Therefore, neither maximizing available cores or threads nor relying on the V-Cache CCD is consistently optimal, emphasizing the need for effective topology decisions through comprehensive workload characterization.
Large Language Models (LLMs) with tens or even hundreds of billions of parameters have become the foundation of modern AI applications. However, fine-tuning such massive models is severely constrained by the limited GPU memory. Existing memory-saving systems, such as ZeRO-based offloading in DeepSpeed, reduce GPU memory usage but inevitably incur substantial I/O overhead, especially when model states reside on slow storage devices, such as NVMe SSDs. As a result, the memory bottleneck in large-scale fine-tuning is transformed into an I/O bottleneck. Although prior systems have employed strategies like parameter prefetching and partial asynchronous execution, they remain limited by synchronous I/O-communication dependencies and the lack of fine-grained read/write I/O scheduling. To address these limitations, we propose IOC, an I/O Coordination Optimization framework that maximizes pipeline parallelism across different phases of LLM fine-tuning. IOC introduces three key mechanisms: (1) An All-Gather prefetching technique based on an I/O state hash table, which completely decouples All-Gather prefetching from parameter I/O, achieving continuous overlap among I/O, communication, and computation; (2) The parameter update phase is refactored into an asynchronous pipeline with explicit I/O isolation, where the optimizer state write-back is executed in a semi-asynchronous manner, thereby mitigating read/write contention and reducing synchronization stalls; (3) Multi-disk parallelism is leveraged by introducing an additional disk to further relieve I/O contention and defer synchronization waits to the latest possible time point. Experimental results demonstrate that IOC significantly accelerates LLM finetuning while preserving low memory consumption. The end-toend fine-tuning time on the Llama-70B model is reduced by $\mathbf{2 1. 5 \%}$ and 34.3% in single-disk and multi-disk configurations compared to the baseline.
As conventional CMOS technology approaches power and performance limits, superconducting single flux quantum (SFQ) logic offers a path to high-speed, energy-efficient computing. However, SFQ circuits require cryogenic temperatures, introducing complex challenges in memory integration and data movement between thermal zones. This paper presents an end-to-end simulation framework for hybrid SFQ/CMOS-memory systems that accurately models processor, memory, and interconnect behavior across cryogenic $(4 \mathrm{~K}, 77 \mathrm{~K})$ and room temperatures $(300 \mathrm{~K})$. The framework integrates gate-level pipelined Rapid SFQ (RSFQ) RISC-V processors, temperature-aware CryoMEM memory models, and physically grounded interconnect latency models. The simulator facilitates cross-layer design space exploration across diverse parameters such as cache placement, interconnect stack selection, and granularity. These features allow the community to identify technological gaps and re-evaluate the bottlenecks in memory-compute throughput. Our evaluations highlight the critical interplay between processor frequency and memory bandwidth, demonstrate the speedup potential of 4K SFQ caches, and quantify the impact of cryostat cabling choices on system performance.
Branch prediction in superscalar processors faces a fundamental trade-off between prediction latency and accuracy. Ideally, a designer would like a single, very large predictor that can make multiple high-quality predictions in one cycle, but such a design only works at low frequency. Traditional multi-level or override predictors provide fast restart after mispredictions but sacrifice accuracy and effective capacity due to redundancy between the slow and fast predictors. Ahead predictors tolerate prediction latency by using older history as the predictor index. The reduced history affects accuracy, and the large predictor affects restart latency. This latency–accuracy tension limits performance in wide-issue superscalar designs. We propose a biased overriding multi-level hierarchy with three predictor levels (L0, L1, L2). L0 uses a High-Confidence-Only Taken (HOTP) predictor that only predicts high-confidence taken control-flow instructions. L1 and L2 are TAGE-based predictors sharing a bimodal table with biased updates. This design achieves fast 1-cycle restart for common branches while maintaining high accuracy for difficult predictions. Our evaluation shows that HOTP $+\mathrm{L} 1+\mathrm{L} 2$ achieves $3.4 \%$ higher IPC than TAHEAD and $3.2 \%$ higher than a traditional 2bit $+\mathrm{L} 1+\mathrm{L} 2 \mathrm{nb}$ hierarchy, demonstrating that properly designed multi-level hierarchies can surpass continuous-prediction approaches. For superscalar implementations requiring multiple predictions per cycle, we size a superscalar TAGE variant (BATAGE) using Optuna-based search with varying table sizes and aspect ratios. Combining HOTP, biased training, and this optimally sized predictor yields $3.7 \%$ IPC improvement over traditional 3-level predictors (2bit+L1+L2nb), offering concrete design options for modern open-source superscalar cores.
As core counts expand through chiplet integration, advanced packaging, and memory expansion technologies, modern systems increasingly rely on large-scale and coherent shared-memory architectures. While coherence protocols are well studied, the empirical behavior of coherent data movement at system scale remains poorly understood. This work presents a detailed characterization of coherent data movement using cyclelevel full-system simulation of a modeled large-scale hierarchical shared-memory system. Across representative HPC and graph processing workloads, we find that coherence overheads are dominated by a small subset of high-impact pages, despite most pages exhibiting limited sharing. These pages combine wide socket span, high access frequency, and millisecond-scale temporal persistence, generating sustained long-distance coherence traffic that frequently propagates beyond the local chassis into higher levels of the interconnect. Together, these observations indicate that coherence cost and challenges are driven by a sparse but dominant tail of shared data, motivating architectural mechanisms that adapt data placement and coherence anchoring to page-level access behavior to make coherence mechanisms more efficient.
Hardware firewalls are critical components of today’s data centers and large enterprises. However, these firewalls demonstrate poor utilization for network traffic that is dominated by a small number of large sessions (elephant flows), sessions with high network flow bandwidth. For these large sessions (and, in fact, all sessions), we observe that the bottleneck is that the processing of all packets is serialized on a single data processing card (DPC) of the firewall to support per-connection consistency, lowering overall utilization. We make a novel observation that only the stateful inspection phase of packet processing truly needs to be serialized – the content inspection phase of packet processing, which dominates the overall processing time, can be parallelized across multiple DPCs without impacting correctness. Based on this observation, we propose CCPA, a novel architecture of hardware firewalls where the stateful inspection for all packets in a session is first performed sequentially on a dedicated processor before the packets are sent to DPCs for concurrent content inspection. By addressing the utilization bottleneck, CCPA improves the average firewall throughput by 4.29x - 14.3x when using an optical backplane.
Maximizing utilization of server components in data centers (through colocation of applications), while guaranteeing Quality of Service (QoS) for every application is of the major goals in enterprise environments. To ensure proper QoS, tools that measure a device utilization play a major role. Such tools report the capacity (i.e., number and type of applications) that may run on each device in real-time. We observe that the publicly-available tools to measure storage device utilization are very limited, and their utilization measurement logic is designed assuming traditional storage devices (with limited in-flight I/O requests). We reveal that the latest version of these tools such as Linux iostat have significant disparity with real utilization of emerging high-performance devices such as Persistent Memory or modern NVMe SSDs.In this paper, we first propose True Utilization metric for storage devices, which is the ratio of real-time I/O per Seconds (IOPS) to its saturation IOPS under same workload patterns. Second, we propose a framework that (a) explores different linear and non-linear models to predict saturation IOPS of each storage device under different workload patterns, (b) combines real-time IOPS measurements with our trained model outputs to estimate True Utilization accurately. We evaluate our proposed framework on different types of SSDs and persistent memories, and show it can accurately calculate True Utilization in real-time, and only have a negligible one-time training overhead.
Energy efficiency is a first-order concern for mobile devices. Although garbage collection is central to the Android platform, the energy costs of garbage collection are not well understood. This is likely due to major methodological hurdles: first, heterogeneous hardware and inherently multi-tenanted software; second, UI-induced waits confound classic performance analysis methodologies; and third, difficulty in measuring energy, which is often not supported by the device, and if it is, has low temporal resolution. These hurdles have clouded our understanding of the energy costs of GC, and may have led to missed opportunities for more energy efficient mobile software. The principal contributions of this paper are new methodologies and a study of the energy costs of garbage collection that uses them. We introduce: i) new insight into the deep problems that UI interactions bring to the performance analysis of Android applications; ii) methodologies that address the problem of cost attribution when the resolution of the measure is coarser than the events being studied; iii) methodologies that control for the hardware heterogeneity found on modern devices; iv) a methodology that emulates the multi-tenanted environment found on mobile devices; and v) an implementation that allows us to quantify the tradeoff due to parallelism in GC. We perform these experiments using a suite of real-world applications, a subset of the DaCapo benchmarks, and a microbenchmark. We show: i) that waits introduced by UI interactions make performance analysis of Android applications deeply challenging; ii) that single-threaded GC typically outperforms parallel GC on mobile devices; iii) that garbage collection has a significant energy overhead on Android applications and is sensitive to GC algorithm and configuration; iv) that different subsystems’ energy consumption is affected by GC choice and configuration. This work sheds new light on energy overheads on mobile devices and we hope may motivate new designs in this impactful domain.
Deep neural network (DNN) inference relies heavily on Tensor Core operations, which are vulnerable to transient hardware faults in computation pipelines not protected by errorcorrecting codes (ECC). Prior fault injection work has explored both application-level and instruction-level effects on DNN accuracy. However, existing application-level approaches support only coarse perturbations and do not capture hardware execution details, while instruction-level approaches lack application-level context.To address this gap, we propose TensorDynamic, an application-aware instruction-level dynamic fault injection tool for Tensor Core execution in DNN workloads. TensorDynamic enables fine-grained fault injection into MMA (matrix-multiplyaccumulate) instructions during DNN execution. Across multiple models, we show that, under the same error injection rate and severity, application-level fault injection can produce substantially different inference outcomes from instruction-level fault injection. This result underscores the need for execution-aware fault injection when evaluating DNN resilience on GPU Tensor Cores.
