On-device LLM inference is attractive for privacy and responsiveness, but remains challenging on mobile and embedded devices because model weights far exceed available DRAM. Prior systems exploit activation sparsity and offload weights to SSD or flash storage, but face a fundamental systems trade-off: accurate sparse execution decisions require the latest context, whereas efficient computation-I/O overlap requires early prediction. As a result, existing designs either serialize execution or incur redundant weight fetches, extra computation, and large cache overheads. We present LeanStream, a streaming speculate-and-refine framework for efficient on-device LLM inference. LeanStream progressively refines computation, loading, and cache-retention priorities using partial GPU results, enabling fine-grained overlap between GPU execution and storage I/O. We implement LeanStream on both mobile and embedded platforms. Compared with prior on-device LLM inference systems, LeanStream reduces memory usage by 4.8× to 7.5× at the best throughput achieved by prior work, while further improving token generation throughput by 1.6× to 2.1×.

As an increasing number of embedded platforms incorporate multiple processing units, shared resource contention induced unpredictable execution time poses a challenge for real-time system design. Memory bandwidth regulation is a popular mitigation approach, and MemGuard is the canonical example. Recently, MemPol introduced a new bandwidth regulation mechanism, which was compared with MemGuard. Specifically, they reported significant overheads for MemGuard, citing up to a 1.79x slowdown, to contextualize MemPol's comparative benefits. This report is meant to clarify and add the necessary nuance to the experiments carried out in that prior work. Specifically, we show that the MemGuard overheads presented in these prior evaluations were unintentionally amplified as the result of using a suboptimal configuration with an older version of MemGuard, wherein the benchmark under test was pinned directly to the master core responsible for handling global timer interrupts. By faithfully reproducing these specific experiments using a modern, decentralized implementation of MemGuard, we demonstrate that the actual execution overhead drops significantly under identical conditions. Consequently, when evaluated with a properly configured recent version, MemGuard exhibits an overhead that is highly comparable to MemPol's overhead. By revisiting these baseline metrics, this report provides an updated and comprehensive perspective required for future evaluations of memory bandwidth regulators.

The shift to data-intensive processing from the cloud to the edge has introduced new challenges and expectations for the next generation of intelligent computing systems. As the memory wall continues to grow, modern systems can only meet these performance expectations by displaying data access patterns that exhibit ideal layouts in memory and ideal spatiotemporal locality in caches. However, only a few data-intensive applications are characterized by ideal locality. Instead, most applications exhibit either (i) poor locality when naively implemented and must undergo costly redesigns and tuning or (ii) inflated memory footprint to offer proper locality. To address the aforementioned challenges, we propose a hardware/software co-designed approach that can be implemented on commercially available SoC/FPGA platforms. Our approach seamlessly inserts in the CPUs' data path a Tensor Memory Engine that provides data with an ideal cache locality to running applications by (i) accessing the memory on behalf of the CPUs and (ii) composing a re-organized view of the memory layout. Unlike in- and near-memory computing approaches, it sets itself apart by clearly decoupling computing and memory accesses; computation is still performed on CPUs while the data re-organization is delegated to the Tensor Memory Engine.

Latency-accuracy tradeoffs are fundamental in real-time applications of deep neural networks (DNNs) for cyber-physical systems. In autonomous driving, in particular, safety depends on both prediction quality and the end-to-end delay from sensing to actuation. We observe that (1) when latency is accounted for, the latency-optimal network configuration varies with scene context and compute availability; and (2) a single fixed-resolution model becomes suboptimal as conditions change. We present a multi-resolution, end-to-end deep neural network for the CARLA urban driving challenge using monocular camera input. Our approach employs a convolutional neural network (CNN) that supports multiple input resolutions through per-resolution batch normalization, enabling runtime selection of an ideal input scale under a latency budget, as well as resolution retargeting, which allows multi-resolution training without access to the original training dataset. We implement and evaluate our multi-resolution end-to-end CNN in CARLA to explore the latency-safety frontier. Results show consistent improvements in per-route safety metrics - lane invasions, red-light infractions, and collisions - relative to fixed-resolution baselines.

Modern multicore system-on-chips (SoCs) share off-chip DRAM across cores, where bank-level interference can significantly degrade performance and threaten real-time guarantees. While prior work has focused on per-core bandwidth regulation, these approaches treat main memory as a monolithic resource and overlook DRAM's inherent bank-level parallelism. We show that DRAM interference is fundamentally a bank-level phenomenon. We characterize the guaranteed bandwidth of modern DRAM, demonstrate that it remains effectively constant across generations, and show how this limitation can be exploited by single-bank attacks. These results highlight the need for bank-aware memory management for predictable and efficient real-time systems. We design and implement a novel per-bank memory bandwidth regulator in an open-source RISC-V SoC and evaluate it using FireSim with both synthetic and real-world workloads. Our evaluation demonstrates that per-bank regulation effectively mitigates adversarial bank contention and achieves a 5.74x average throughput improvement for best-effort workloads over traditional bank-oblivious approaches while providing the same-level of performance isolation guarantees for real-time workloads.

Making tradeoffs between execution latency and result utility (i.e., anytime computing) to adapt to dynamic operational requirements has been shown to enhance the performance of cyber-physical systems. In this work, we focus on enabling anytime computing for deep neural networks (DNNs) that process LiDAR point clouds for 3D object detection. We propose a novel method that enables multi-resolution inference, allowing the input to be dynamically scaled and processed at the resolution needed to meet timing requirements. Importantly, our memory-efficient approach requires the deployment of only a single DNN model, avoiding the need to deploy multiple models, each trained for a different input resolution. We also introduce a deadline-aware scheduler that selects the highest possible resolution for any given input by accurately predicting the execution time for all possible resolutions at runtime, which is a challenging task due to the irregularity of LiDAR point clouds. Experimental results on the nuScenes autonomous driving dataset demonstrate that our method significantly outperforms existing anytime computing approaches for LiDAR object detection. By achieving superior detection accuracy under varying time constraints while maintaining deployment simplicity, our work establishes a new state-of-the-art in this domain.

Recent advancements in on-device training for deep neural networks have underscored the critical need for efficient activation compression to overcome the memory constraints of mobile and edge devices. As activations dominate memory usage during training and are essential for gradient computation, compressing them without compromising accuracy remains a key research challenge. While existing methods for dynamic activation quantization promise theoretical memory savings, their practical deployment is impeded by system-level challenges such as computational overhead and memory fragmentation. To address these challenges, we introduce DAF, a Dynamic Activation Framework that enables scalable and efficient on-device training through system-level optimizations. DAF achieves both memory- and time-efficient dynamic quantization training by addressing key system bottlenecks. It develops hybrid reduction operations tailored to the memory hierarchies of mobile and edge SoCs, leverages collaborative CPU-GPU bit-packing for efficient dynamic quantization, and implements an importance-aware paging memory management scheme to reduce fragmentation and support dynamic memory adjustments. These optimizations collectively enable DAF to achieve substantial memory savings and speedup without compromising model training accuracy. Evaluations on various deep learning models across embedded and mobile platforms demonstrate up to a 22.9× reduction in memory usage and a 3.2× speedup, making DAF a scalable and practical solution for resource-constrained environments.

Urban Air Mobility (UAM) applications, such as air taxis, will rely heavily on perception for situational awareness and safe operation. With recent advances in AUML, state-of-the-art perception systems can provide the high-fidelity information necessary for UAM systems. However, due to size, weight, power, and cost (SWaP-C) constraints, the available computing resources of the on-board computing platform in such UAM systems are limited. Therefore, real-time processing of sophisticated perception algorithms, along with guidance, navigation, and control (GNC) functions in a UAM system, is challenging and requires the careful allocation of computing resources. Furthermore, the optimal allocation of computing resources may change over time depending on the speed of the vehicle, environmental complexities, and other factors. For instance, a fast-moving air vehicle at low altitude would need a low-latency perception system, as a long delay in perception can negatively affect safety. Conversely, a slowly landing air vehicle in a complex urban environment would prefer a highly accurate perception system, even if it takes a little longer. However, most perception and control systems are not designed to support such dynamic reconfigurations necessary to maximize performance and safety. We advocate for developing "anytime" perception and control capabilities that can dynamically reconfigure the capabilities of perception and GNC algorithms at runtime to enable safe and intelligent UAM applications. The anytime approach will efficiently allocate the limited computing resources in ways that maximize mission success and ensure safety. The anytime capability is also valuable in the context of distributed sensing, enabling the efficient sharing of perception information across multiple sensor modalities between the nodes.

This work addresses the challenge of adapting dynamic deadline requirements for the LiDAR object detection deep neural networks (DNNs). The computing latency of object detection is critically important to ensure safe and efficient navigation. However, the state-of-the-art LiDAR object detection DNNs often exhibit significant latency, hindering their real-time performance on the resource-constrained edge platforms. Therefore, a tradeoff between the detection accuracy and latency should be dynamically managed at runtime to achieve the optimum results. In this article, we introduce versatile anytime algorithm for the LiDAR Object detection (VALO), a novel data-centric approach that enables anytime computing of 3-D LiDAR object detection DNNs. VALO employs a deadline-aware scheduler to selectively process the input regions, making execution time and accuracy tradeoffs without architectural modifications. Additionally, it leverages efficient forecasting of the past detection results to mitigate possible loss of accuracy due to partial processing of input. Finally, it utilizes a novel input reduction technique within its detection heads to significantly accelerate the execution without sacrificing accuracy. We implement VALO on the state-of-the-art 3-D LiDAR object detection networks, namely CenterPoint and VoxelNext, and demonstrate its dynamic adaptability to a wide range of time constraints while achieving higher accuracy than the prior state-of-the-art. Code is available at https://github.com/CSL-KU/VALOgithub.com/CSL-KU/VALO .

In this paper, we present a solution to the industrial challenge put forth by ARM in 2022. We systematically analyze the effect of shared resource contention to an augmented reality head-up display (AR-HUD) case-study application of the industrial challenge on a heterogeneous multicore platform, NVIDIA Jetson Nano. We configure the AR-HUD application such that it can process incoming image frames in real-time at 20Hz on the platform. We use Microarchitectural Denial-of-Service (DoS) attacks as aggressor workloads of the challenge and show that they can dramatically impact the latency and accuracy of the AR-HUD application. This results in significant deviations of the estimated trajectories from known ground truths, despite our best effort to mitigate their influence by using cache partitioning and real-time scheduling of the AR-HUD application. To address the challenge, we propose RT-Gang++, a partitioned real-time gang scheduling framework with last-level cache (LLC) and integrated GPU bandwidth throttling capabilities. By applying RT-Gang++, we are able to achieve desired level of performance of the AR-HUD application even in the presence of fully loaded aggressor tasks.

Prior research has demonstrated the effectiveness of end-to-end deep learning for robotic navigation, where the control signals are directly derived from raw sensory data. However, the majority of existing end-to-end navigation solutions are predominantly camera-based. In this paper, we introduce TinyLidarNet, a lightweight 2D LiDAR-based end-to-end deep learning model for autonomous racing. An F1TENTH vehicle using TinyLidarNet won 3rd place in the 12th F1TENTH Autonomous Grand Prix competition, demonstrating its competitive performance. We systematically analyze its performance on untrained tracks and computing requirements for real-time processing. We find that TinyLidarNet’s 1D Convolutional Neural Network (CNN) based architecture significantly outperforms widely used Multi-Layer Perceptron (MLP) based architecture. In addition, we show that it can be processed in real-time on low-end micro-controller units (MCUs).

Modern commercial-off-the-shelf (COTS) multicore processors have advanced memory hierarchies that enhance memory-level parallelism (MLP), which is crucial for high performance. To support high MLP, shared last-level caches (LLCs) are divided into multiple banks, allowing parallel access. However, uneven distribution of cache requests from the cores, especially when requests from multiple cores are concentrated on a single bank, can result in significant contention affecting all cores that access the cache. Such cache bank contention can even be maliciously induced-known as cache bank-aware denial-of-service (DoS) attacks-in order to jeopardize the system’s timing predictability. In this paper, we propose a per-bank bandwidth regulation approach for multi-banked shared LLC based multicore realtime systems. By regulating bandwidth on a per-bank basis, the approach aims to prevent unnecessary throttling of cache accesses to non-contended banks, thus improving overall performance (throughput) without compromising isolation benefits of throttling. We implement our approach on a RISC-V system-on-chip (SoC) platform using FireSim and evaluate extensively using both synthetic and real-world workloads. Our evaluation results show that the proposed per-bank regulation approach effectively protects real-time tasks from co-running cache bank-aware DoS attacks, and offers up to a $3.66 \times$ performance improvement for the throttled benign best-effort tasks compared to prior bank-oblivious bandwidth throttling approaches.

The paper discusses algorithmic priority inversion in mission-critical machine inference pipelines used in modern neural-network-based perception subsystems and describes a solution to mitigate its effect. In general, priority inversion occurs in computing systems when computations that are "less important" are performed together with or ahead of those that are "more important." Significant priority inversion occurs in existing machine inference pipelines when they do not differentiate between critical and less critical data. We describe a framework to resolve this problem and demonstrate that it improves a perception system's ability to react to critical inputs, while at the same time reducing platform cost.

Contention for shared microarchitectural resources is a significant challenge for using modern heterogeneous multicore processors in safety-critical real-time applications. In this talk, I will first discuss major microarchitectural interference channels in modern multicore architecture. I will describe why microarchitectural designs aiming to achieve high memory-level parallelism (MLP), such as non-blocking cache and banked cache/DRAM organizations, could also become problematic interference channels from a real-time perspective. To illustrate their potential dangers, I will explain several effective “attack” strategies that have been shown to cause massive cross-core interference in real multicore platforms. Next, I will focus on the industrial challenge put forth by ARM in 2022, which is based on an augmented reality head-up display case-study from the automotive domain. I will discuss the impact of “aggressor” workloads on the performance of the case study's real-time applications on a real heterogeneous multicore platform (Jetson Nano). I will then present our software-based solution to achieve desired real-time performance without resorting to over-provisioning. Lastly, I will discuss the limitations of our approach and the need for better hardware support.

Unpredictable variation in execution times due to contention in shared hardware resources in integrated CPU-GPU multicore platforms remains a major challenge for safety-critical real-time systems. We present BandWatch, a system-wide memory bandwidth regulation system, which dynamically regulates both CPU cores and the GPU to protect the performance of critical real-time tasks while minimizing the negative throughput impact to the throttled non real-time tasks. BandWatch is implemented on Linux as a kernel module on a NVIDIA Jetson Nano platform. The extensive evaluation results using both real-world and synthetic workloads show the effectiveness of BandWatch.

In this paper, we identify that bank contention in the shared last-level cache (LLC) of multicore processors can cause significant execution time slowdown to cross-core victims even when the cache is partitioned. We propose a Cache Bank-Aware Denial-of-Service (DoS) attack, which is specially engineered to induce bank contention. This is accomplished by leveraging publicly available cache bank mapping information on commercial off-the-shelf (COTS) ARM multicore processors to generate a large number of concurrent memory accesses to a particular cache bank. Importantly, this attack can be mounted from the user-space to delay cross-core victims without requiring any special privilege or OS level support. We implement and evaluate the proposed DoS attack on two ARM multicore platforms using both synthetic and real-world workloads as victim tasks. The results show that the proposed Cache Bank-Aware DoS attack causes up to 9. 7X slowdown in synthetic workloads and 2. 3X slowdown in real-world workloads even when stateof-the-art isolation techniques, namely LLC partitioning and DRAM bandwidth throttling, are used to protect the victim. In other words, the proposed attack can effectively bypass existing DoS attack mitigation techniques. We identify LLC bandwidth throttling and LLC bank partitioning as potential mitigation solutions and discuss their limitations.

Transient execution attacks use microarchitectural covert channels to leak secrets that should not have been accessible during logical program execution. Commonly used micro-architectural covert channels are those that leave lasting footprints in the micro-architectural state, for example, a cache state change, from which the secret is recovered after the transient execution is completed. In this paper, we present SpectreRewind, a new approach to create and exploit contention-based covert channels for transient execution attacks. In our approach, a covert channel is established by issuing the necessary instructions logically before the transiently executed victim code. Unlike prior contention-based covert channels, which require simultaneous multi-threading (SMT), SpectreRewind works on a single hardware thread and does not require SMT. We show that contention on the floating point division unit on commodity out-of-order processors can be used to create a high-performance ( ∼ 100 KB/s), low-noise covert channel for transient execution attacks instead of commonly used flush+reload-based cache covert channels. We also show that the proposed covert channel works in the JavaScript sandbox environment of a Chrome browser and can be used in a Meltdown attack.

In this work, we present a novel scheduling frame-work enabling anytime perception for deep neural network (DNN) based 3D object detection pipelines. We focus on computationally expensive region proposal network (RPN) and per-category multi-head detector components, which are common in 3D object detection pipelines, and make them deadline-aware. We propose a scheduling algorithm, which intelligently selects the subset of the components to make effective time and accuracy trade-off on the fly. We minimize accuracy loss of skipping some of the neural network sub-components by projecting previously detected objects onto the current scene through estimations. We apply our approach to a state-of-art 3D object detection network, PointPillars, and evaluate its performance on Jetson Xavier AGX using nuScenes dataset. Compared to the baselines, our approach significantly improve the network’s accuracy under various deadline constraints.

In this paper, we study the effectiveness of denial-of-service (DoS) attacks on Intel’s heterogeneous multicore system-on-chips with integrated GPU (iGPU) in which the last level cache (LLC) and the main memory subsystem are shared between the multicore CPU and the iGPU. Using two Intel processors with iGPU, we evaluate four different DoS attacks, three CPU based and one iGPU based, and show they can induce very high degree of shared resource contention and thus dramatically slowdown the victim’s execution time. We further evaluate the effectiveness of Intel’s recent hardware based shared resource isolation mechanisms, namely Intel Cache Allocation Technology (CAT) and Graphics Technology Class of Service (GT COS), which provide shared LLC partitioning capability for the CPU cores and the iGPU, respectively, in defending against these DoS attacks. Using both synthetic and real-world benchmarks, we find that hardware based LLC partitioning mechanisms does provide spatial LLC space isolation but does not necessarily provide temporal isolation.