Deep Learning (DL) models deployed on edge inference accelerators must meet strict power efficiency targets. While pruning and quantization effectively reduce idle power, active power, particularly from weight switching in Multiply-Accumulate (MAC) operations, can contribute up to 15-20% of total power and remains underexplored. To address this, we present DYNNER, a novel compiler-level weight-reordering technique that analyzes bit-level switching patterns in Processing Elements (PEs) and reorganizes weights to minimize Hamming distance during MAC execution. To our knowledge, this is the first approach that reduces switching-induced active power without any hardware changes or accuracy loss. DYNNER generalizes beyond output-stationary designs to a broad class of accelerators with deterministic operand streaming, and formalizes two deployment schemes, Scheme-1 (K+C reorder, skip-layer) and Scheme-2 (C-only reorder, no skip), together with a compiler-level selector that chooses the best scheme per model. DYNNER achieves up to 50% reduction in layer-level switching activity, resulting in 6% total active power savings at the accelerator level, while preserving the original network topology and weights. Beyond this lossless mode, we introduce a lossy extension wherein DYNNER is combined with low-bit precision quantization, not for traditional compression or speedup, but, for the first time, as a mechanism to reduce switching power by concentrating the weight-value distribution. This integrated approach achieves up to 43% additional switching savings and enables a tunable accuracy-power trade-off. Building on insights from DYNNER, we also propose DynArch, a switching-aware architecture frame-work that explores MAC-to-PE allocation strategies. DynArch favors higher MAC density per PE and lower PE parallelism, naturally exposing weight locality and reducing cross-filter toggling, yielding up to 6% active power savings for individual layers, even before reordering is applied. Together, DYNNER and DynArch introduce a new class of switching-centric optimizations, enabling fine-grained, power-aware design for accelerators at the edge.

Transformer neural networks have emerged as the state-of-the-art in AI across text, audio, image, and video processing tasks. However, the attention mechanism that is core to Transformers has a complexity that scales quadratically with input sequence length, making it a computational bottleneck. The Softmax operation within the attention computation is in turn a significant reason for this bottleneck. We identify three primary reasons for the computational challenge of Softmax: (1) the intrinsic complexity of the exponentiation and division operations, (2) the data dependencies among operations within Softmax, which limit parallelism, and (3) reduction operations such as finding the maximum, which do not map well to NPUs or MAC Arrays that are designed for matrix multiply operations. Existing approaches to improve the computational efficiency of Softmax often require extensive pretraining or fine-tuning, or specialized hardware. In contrast, we propose Softprox, a post-fine-tuning technique that can accelerate Softmax computations on commodity hardware. Softprox utilizes three complementary approximations - AttSkip, SoftNet, and StatMax - to optimize Softmax computations in any given fine-tuned network. AttSkip is an attention score reuse mechanism that minimizes the required attention computations by reusing Softmax scores from the previous layer. For layers and heads that cannot adopt AttSkip without accuracy degradation, SoftNet employs a low-complexity neural network to approximate the Softmax operation, reducing reliance on complex exponentiation and division operations. Finally, StatMax replaces the dynamic search for a vector's maximum element with a statistically pre-computed maximum value. Across a suite of Transformer models for text, audio, and vision tasks, Softprox demonstrates an average runtime inference speedup of 21.72%, with a maximum speedup of 40.22% on the Intel (R) Core (TM) Ultra Series 1 NPU.

The rise of on-device AI has transformed modern client PCs into powerful platforms for executing diverse AI workloads locally, enabling low-latency, energy-efficient, and privacy-preserving applications. Traditional on-device AI applications have predominantly focused on computer vision tasks such as image classification, segmentation, and object detection, enabled by Convolutional Neural Networks (CNNs) and widely explored on edge devices. However, the recent surge in Generative AI and Large Language Models (LLMs) has expanded the application landscape to include personal assistants, contextual search, knowledge graph reasoning, and real-time transcription. To support these emerging workloads, the community has largely adopted transformer-based architectures due to their versatility and strong modeling capabilities across modalities. Despite their success, transformers exhibit quadratic complexity in sequence length, making them inefficient for long-context tasks. As a result, State Space Models (SSMs) have gained traction as efficient alternatives, offering linear or near-linear scaling with competitive accuracy for applications such as transcription, translation, and long-sequence language modeling. Meanwhile, Graph Neural Networks (GNNs) have emerged as state-of-the-art models for tasks involving structured and relational data, including knowledge graph reasoning, personalized recommendations, and event-driven visual analytics, achieving superior performance compared to transformers in certain domains. While CNN and transformer workloads are already well-supported on edge platforms, the enablement and optimization of newer model classes remain underexplored. To address this, we present NEXUS , a unified framework that establishes a systematic methodology for accelerating emerging AI workloads on commercial-off-the-shelf (COTS) Neural Processing Units (NPUs). NEXUS introduces a general systematic three-step approach: (1) Enablement , which adapts model execution to NPUs’ static input shapes, partitioned compute, and memory constraints; (2) Optimization , which remaps control-heavy and sequential operations to hardware-efficient regular data-parallel kernels such as matrix multiplications; and (3) Quality-Efficiency Tradeoffs , which apply quantization and approximations to improve performance and energy efficiency with minimal accuracy loss. While this work focuses on Graph Neural Networks (GNNs) and State Space Models (SSMs) as representative case studies, the NEXUS methodology is model-agnostic and can be readily extended to future emerging workloads and NPU platforms. Experimental evaluations on Intel® Core™ Ultra Series 1 and 2 AI PCs demonstrate that NEXUS achieves up to 10.8 × speedup and 8.6 × energy efficiency improvement for GNNs over CPU baselines, and up to 3.8 × latency reduction for SSMs. Individual techniques like CumBA, ReduBA, ActiBA, and QuBA contribute 2.7 ×, 1.2 ×, 3.7 ×, and 1.7 × reductions, respectively, all while maintaining model accuracy within 1.5% of full-precision baselines. Our code implementation is available at https://github.com/arghadippurdue/NEXUS.

Hardware verification is crucial in modern SoC design, consuming around 70 of development time. SystemVerilog assertions ensure correct functionality. However, existing industrial practices rely on manual efforts for assertion generation, which becomes increasingly untenable as hardware systems become complex. Recent research shows that Large Language Models (LLMs) can automate this process. However, proprietary SOTA models like GPT-4o often generate inaccurate assertions and require expensive licenses, while smaller open-source LLMs need fine-tuning to manage HDL code complexities. To address these issues, we introduce **VERT**, an open-source dataset designed to enhance SystemVerilog assertion generation using LLMs. VERT enables researchers in academia and industry to fine-tune open-source models, outperforming larger proprietary ones in both accuracy and efficiency while ensuring data privacy through local fine-tuning and eliminating costly licenses. The dataset is curated by systematically augmenting variables from open-source HDL repositories to generate synthetic code snippets paired with corresponding assertions. Experimental results demonstrate that fine-tuned models like Deepseek Coder 6.7B and Llama 3.1 8B outperform GPT-4o, achieving up to 96.88 over base models and 24.14 OpenPiton and Pulpissimo. VERT is available at https://github.com/AnandMenon12/VERT.

Transformers underpin many on-device AI tasks, yet quadratic attention and large MLP blocks limit long-context inference on resource-constrained edge systems. Recent alternatives—Hyena (implicit long convolutions with gating) and Kolmogorov–Arnold Networks (KANs; spline-based MLP replacements)—promise algorithmic gains in scalability and interpretability, but their hardware efficiency on edge accelerators remains unclear. We present HKN, a systematic evaluation of Hyena and KAN on the AI compute engines of an edge-class SoC: CPU, GPU, and Neural Processing Unit (NPU). Using a unified software stack and measurement methodology, HKN profiles end-to-end latency, throughput, and energy, while comparing these modules directly against the attention and MLP components they are designed to supersede. The analysis focuses primarily on the NPU, with results on CPU and GPU included to confirm broader impact. At the operator level, we identify how different classes of operations—dense matrix multiplications, elementwise functions, and basis-function computations—map onto the NPU's MAC-centric datapath versus DSP subsystem, exposing mismatches between algorithmic structure and hardware specialization. Across all compute engines, Hyena and KAN exhibit higher latency and energy than their baselines, with the gap driven by memory-bound long convolutions in Hyena and recursive spline evaluations in KAN. By locating these inefficiencies, HKN highlights the challenges of utilizing existing AI accelerators effectively and provides a foundation for software–hardware co-design to enable practical deployment of long-context Transformer alternatives on edge devices.

Graph Neural Networks (GNNs) are vital for learning from graph-structured data, enabling applications in network analysis, recommendation systems, and speech analytics. Deploying them on edge devices like client PCs and laptops enhances real-time processing, privacy, and cloud independence. GNNs aid Retrieval-Augmented Generation (RAG) for Large Language Models (LLMs) and enable event-based vision tasks. However, irregular memory access, sparsity, and dynamic structures cause high latency and energy overhead on resource-constrained devices. While modern edge processors integrate CPUs, GPUs, and NPUs, NPUs designed for data-parallel tasks struggle with irregular GNN computations. We introduce GraNNite, the first hardware-aware framework optimizing GNN execution on commercial-off-the-shelf (COTS) SOTA DNN accelerators via a structured three-step methodology: (1) enabling NPU execution, (2) optimizing performance, and (3) trading accuracy for efficiency gains. Step 1 employs GraphSplit for workload distribution and StaGr for static aggregation, while GrAd and NodePad handle dynamic graphs. Step 2 boosts performance using EffOp for control-heavy tasks and GraSp for sparsity exploitation. Graph Convolution optimizations PreG, SymG, and CacheG reduce redundancy and memory transfers. Step 3 balances quality versus efficiency, where QuantGr applies INT8 quantization, and GrAx1, GrAx2, and GrAx3 accelerate attention, broadcast-add, and SAGE-max aggregation. On Intel Core Ultra AI PCs, GraNNite achieves 2.6X to 7.6X speedups over default NPU mappings and up to 8.6X energy gains over CPUs and GPUs, delivering 10.8X and 6.7X higher performance than CPUs and GPUs, respectively, across GNN models.

Neural processing units (NPUs) have become essential in modern client and edge platforms, offering unparalleled efficiency by delivering high throughput at low power. This is critical to improve the TOPS/W of the NPU, leading to longer battery life. While NPUs were initially designed to efficiently execute computer vision (CV) workloads such as CNNs, the rising demand to run transformer-based large language models (LLMs) locally now calls for significant architectural and software adaptation. This paper presents LLM-NPU, a comprehensive software-hardware co-optimization framework that enables scalable, power-efficient LLM deployment on NPUs under tight compute and memory budgets. We present software solutions such as vertical and horizontal operator fusion, quantization-aware weight compression, hybrid key-value (KV) quantization, eviction strategies, and static-shape inference that target memory bottlenecks and compute inefficiencies in LLM execution. On the hardware side, we explore domain-specialized NPU enhancements, including processing-in-memory architectures, extended input channel accumulation, structured sparsity acceleration, GEMM engine optimizations, mixed precision, microscaling format support, and fusion-aware execution pipelines. These co-designed innovations can collectively improve the energy, throughput, and latency of NPUs for LLM workloads.

Assertions are critical components used in hardware verification, ensuring robust functionality, fortifying design security, and providing essential verification features. Traditional hardware assertion methods are not automated, complicate security audits, and require effort, causing prolonged development cycles. Recent studies have highlighted the potential of commercial Large Language Models (LLMs) to generate security-focused assertions by leveraging textual data from design specifications. However, reliance on proprietary models like GPT-4 severely jeopardizes IP privacy and data confidentiality, undermining transparency and accountability in data handling practices. In this paper, we address secure hardware assertion generation by proposing a practical approach to significantly enhance the feasibility of open-source LLMs. Our proposed method, OpenAssert, involves fine-tuning existing models to be utilized locally at the user’s end without compromising confidentiality. Additionally, we employ Retrieval Augmentation Generation to refine these models, mitigating hallucinations and security-related errors. OpenAssert demonstrates improvements, achieving up to a 44% increase in rouge-1 score, a 49% improvement in cosine similarity, and a 43.4% reduction in word error rate for security-critical designs compared to open-source models.

In this paper, we propose StruM, a novel structured mixed-precision-based deep learning inference method, co-designed with its associated hardware accelerator (DPU), to address the escalating computational and memory demands of deep learning workloads in data centers and edge applications. Diverging from traditional approaches, our method avoids time-consuming re-training/fine-tuning and specialized hardware access. By leveraging the variance in weight magnitudes within layers, we quantize values within blocks to two different levels, achieving up to a 50 integer weights to 4-bit values across various Convolutional Neural Networks (CNNs) with negligible loss in inference accuracy. To demonstrate efficiency gains by utilizing mixed precision, we implement StruM on top of our in-house FlexNN DNN accelerator [1] that supports low and mixed-precision execution. Experimental results depict that the proposed StruM-based hardware architecture achieves a 31-34 10 configured StruM results in 23-26 savings at the DPU level.

State-Space Models (SSMs) have emerged as efficient alternatives to transformers for sequential data tasks, offering linear or near-linear scalability with sequence length, making them ideal for long-sequence applications in NLP, vision, and edge AI, including real-time transcription, translation, and contextual search. These applications require lightweight, high-performance models for deployment on resource-constrained devices like laptops and PCs. Designing specialized accelerators for every emerging neural network is costly and impractical; instead, optimizing models for existing NPUs in AI PCs provides a scalable solution. To this end, we propose XAMBA, the first framework to enable and optimize SSMs on commercial off-the-shelf (COTS) state-of-the-art (SOTA) NPUs. XAMBA follows a three-step methodology: (1) enabling SSMs on NPUs, (2) optimizing performance to meet KPI requirements, and (3) trading accuracy for additional performance gains. After enabling SSMs on NPUs, XAMBA mitigates key bottlenecks using CumBA and ReduBA, replacing sequential CumSum and ReduceSum operations with matrix-based computations, significantly improving execution speed and memory efficiency. Additionally, ActiBA enhances performance by approximating expensive activation functions (e.g., Swish, Softplus) using piecewise linear mappings, reducing latency with minimal accuracy loss. Evaluations on an Intel Core Ultra Series 2 AI PC show that XAMBA achieves up to 2.6X speed-up over the baseline. Our implementation is available at https://github.com/arghadippurdue/XAMBA.

Deep Neural Networks (DNNs) exhibit sparsity in both activation and weight tensors, but certain layers have higher weight sparsity, while others have higher activation sparsity. This challenges the conventional approach of fixing sparsity acceleration to either weights or activations alone. Conversely, harnessing both-sided sparsity necessitates complex design logic for identifying participating non-zero weights and activation pairs during a multiply-accumulate operation, leading to a significant impact on energy efficiency and area overhead in the edge accelerator. In this paper, we, for the first time, exploit the unbalanced sparsity in DNNs to propose the concept of dynamically Switchable Single-sided Sparsity, SwiSS, to improve energy efficiency in edge DNN accelerators. Through a novel self-adaptive dynamic sparsity selection algorithm, SwiSS can determine whether to enable one-sided weight or one-sided activation sparsity for a sparsity-enabled DNN accelerator. This capability allows SwiSS to dynamically exploit both sides of sparsity while maximizing the associated power and area benefits in the accelerator. Evaluation on state-of-the-art network-dataset configurations conducted on FlexNN [12] accelerator architecture demonstrates that SwiSS yields up to 30.76% and 8.29% improvements in power and area overheads, respectively (which translates to 1.42X and 1.08X improvement in TOPS/W and TOPS/mm2, respectively), compared to a combined two-sided sparsity scenario, with a negligible drop in sparsity acceleration.

This paper introduces FlexNN, a Flexible Neural Network accelerator, which adopts agile design principles to enable versatile dataflows, enhancing energy efficiency. Unlike conventional convolutional neural network accelerator architectures that adhere to fixed dataflows (such as input, weight, output, or row stationary) for transferring activations and weights between storage and compute units, our design revolutionizes by enabling adaptable dataflows of any type through software configurable descriptors. Considering that data movement costs considerably outweigh compute costs from an energy perspective, the flexibility in dataflow allows us to optimize the movement per layer for minimal data transfer and energy consumption, a capability unattainable in fixed dataflow architectures. To further enhance throughput and reduce energy consumption in the FlexNN architecture, we propose a novel sparsity-based acceleration logic that utilizes fine-grained sparsity in both the activation and weight tensors to bypass redundant computations, thus optimizing the convolution engine within the hardware accelerator. Extensive experimental results underscore a significant enhancement in the performance and energy efficiency of FlexNN relative to existing DNN accelerators.

The deployment of computationally intensive workloads, such as Large Language Models (LLMs), at the edge presents significant challenges due to resource constraints and limited computing power. State-of-the-art edge platforms address this with integrated high-performance accelerators optimized for machine learning workloads. While numerous high-performance edge platforms are now available in the market, there is limited exploration of methods to maximize performance and optimize resource utilization at the edge. Our work aims to address this gap by proposing methodologies for maximizing performance through the distribution of model inference on a heterogeneous edge platform. This poster presents preliminary findings from our ongoing research project funded by NSF IUCRC IDEAS Center. The Center is focused on Edge Intelligence and applications that can be potentially mapped to Edge platforms.

Although DNNs have become the AI standard due to algorithmic advancements, their computational and memory demands pose challenges for their deployment on edge devices. Extensive research has explored optimizations to efficiently run DNN models on resource-constrained mobile devices, encompassing both software and hardware enhancements. Sparsity exploitation is a prominent optimization technique that aims to boost DNN inference efficiency and speed by eliminating redundant MAC operations resulting from zero operands. In this paper, we propose HARVEST, a hardware-software co-design approach that utilizes existing sparsity engines in accelerators to introduce sparsity in DNN weights and activations during inference. This technique involves two methods: (1) Activation sparsity is achieved by applying thresholds to intermediate activations. These thresholds are determined based on constraints and statistics of the activations stored in the accelerator’s SRAM banks. This approach applies thresholding to all layer types, including convolution, element-wise, fully connected, attention, normalization layers, and non-linear activation functions, thereby increasing sparsity without any additional overhead in terms of area and power. (2) Weight sparsity is introduced before deployment using customized thresholds for each layer. These methods collectively reduce memory and compute energy consumption, leading to improvements in accelerator energy efficiency, at the cost of minimal accuracy loss. Results on state-of-the-art Transformer and CNN models demonstrate a gain of up to 61% and 80% in activation and weight sparsity, respectively. Exploiting this sparsity, an in-house fully-sparse accelerator provides up to 24%, 36%, and 32% reductions in memory, compute, and overall accelerator energy, respectively, for minimal (< 0.5%) loss in accuracy. Furthermore, HARVEST provides up to 32% and 36% reduction in memory and compute cycle count during DNN inference, leading to increased throughput.

In this paper, we present a novel sparsity-based acceleration logic, that leverages both-sided fine-grained sparsity in the activation and weight tensors to skip ineffectual computations, thereby implementing an efficient convolution engine in a hardware accelerator. However, as demonstrated in this paper, such circuit-level faults, manifested in the sparsity logic can result in graceless degradation in classification accuracy, as well as control failure in the sparse DNN accelerator in mission mode. To circumvent this, we propose RASH, a Reliable deep learning Acceleration framework using Sparsity-based Hardware, that enables in-field detection of faults manifested in the sparsity logic of the DNN accelerator.

State-of-art NPUs are typically architected as a self-contained sub-system with multiple heterogeneous hardware computing modules, and a dataflow-driven programming model. There lacks well-established methodology and tools in the industry to evaluate and compare the performance of NPUs from different architectures. We present an event-based performance modeling framework, VPU-EM, targeting scalable performance evaluation of modern NPUs across diversified AI workloads. The framework adopts high-level event-based system-simulation methodology to abstract away design details for speed, while maintaining hardware pipelining, concurrency and interaction with software task scheduling. It is natively developed in Python and built to interface directly with AI frameworks such as Tensorflow, PyTorch, ONNX and OpenVINO, linking various in-house NPU graph compilers to achieve optimized full model performance. Furthermore, VPU-EM also provides the capability to model power characteristics of NPU in Power-EM mode to enable joint performance/power analysis. Using VPU-EM, we conduct performance/power analysis of models from representative neural network architecture. We demonstrate that even though this framework is developed for Intel VPU, an Intel in-house NPU IP technology, the methodology can be generalized for analysis of modern NPUs.

In the field of artificial intelligence (AI), sensor fusion [1] is becoming an increasingly important technology for many Internet of things (IoT) applications. For intelligent transportation system (ITS), we propose a wireless-assisted automatic online spatial calibration for sensor fusion. Due to the mobility of traffic objects such as vehicles, it is critical to make sure ground truth (GT) position and corresponding sample positions are measured based on raw data collected by different sensors at the same time. We use 5G time sensitive networking (TSN) to achieve this time alignment. Additionally, we illustrate how to perform automatic miscalibration detection and correction based on correctly grouped GT position and sample positions. There are many advantages of using our solution in practical sensor fusion systems that can significantly improve traffic safety and efficiency.

A double-buffered, 4kb standard-cell-based register file with measured 2.4GHz operation at 0.65V, 100°C and scalable performance to 3.7GHz at 0.8V is fabricated in a leading-edge CMOS node. Double-buffering, Gray-coded read/write addressing, super-multi-bit macro standard cells, and mixed-frequency clocking enable a measured peak energy-efficiency of 5.73TOPS/W at 0.5V, 0°C with 14%/11% register file read/write power savings and 28% area reduction over conventional ping-pong design.