Large Language Models (LLMs) have attracted extensive attetion for their remarkable performance across a variety of tasks. However, the considerable computational and memory demands associated with LLMs present challenges for ensuring inference speed performance when deployed in resource-constrained scenarios. Current researches widely adopt model compression techniques and have greatly reduced the computational and memory overhead. Meanwhile, some frameworks that serve as deployment tools for large models can provide optimized deployment strategies based on the characteristics of hardware architectures. In this work, we explored LLM optimization performance on the ARMv9 computing platform, which is mainstream for intelligent edge usage. The inference efficiency was enhanced and memory usage was reduced by lowering dequantization cost, rewriting specific functions in LLaMA.cpp framework and using advanced compilation optimization techniques. Through optimizating and evaluating of the Qwen-1.8B model on ARMv9, inference performance improvement of 23.36x was achieved in decoding stage and 1.68 x in prefilling stage over the baseline with only a 1.04% loss in accuracy. All the developed code in this work is open source at https://github.com/CEATRG/ LLaMA.cpp- arm.
Phase-Change Memory (PCM) based resistive crossbars are a promising technology for energy-efficient neural network inference in analog in-memory computing (AIMC) systems. However, a conductance drift in PCM devices degrades performance over time, posing a challenge for applications requiring long-term stability. In order to overcome such problems, we propose a drift-aware regularization framework that stabilizes weight drift during training, complemented by an attention-based mechanism to prioritize critical features. Our approach significantly improves upon traditional hardware-aware (HWA) training and achieves more robust inference performance in the presence of conductance drift. In a five-year drift simulation, our method reduces classification error to 3.75%, compared to 9.14% with standard HWA training on a benchmark two-layer perceptron for the MNIST dataset. When combined with global drift compensation, the error is further reduced to 2.12%. These results demonstrate the effectiveness of our drift-aware regularization in enhancing the stability and accuracy of neural networks on AIMC hardware, offering a scalable, energy-efficient solution for inference in resource-constrained environments.
While vision transformers excel in various computer vision tasks, their high computational cost limits use on resource-constrained devices, highlighting the need for complexity reduction. In this paper, we present dynamic patch pruning for low complexities of vision transformers (DPP-ViT). To identify relatively more important patches, the column-wise accumulations of attention maps are computed and those are used as importance scores. Through DPP-ViT with block-wise importance score thresholds, our approach considers imagewise difficulties and block-wise sensitivities, removing the patches that contribute the least to accuracies. Additionally, we present a reconfigurable accelerator that dynamically changes dataflow and PE structure by applying pruning-aware row-level reconfiguration. DPP-ViT achieves 47% computation reduction with a minor -0.25% degradation on DeiT-B model in ImageNet top-1 accuracy. The proposed reconfigurable accelerator also achieves 47.96x/ 4.36x/ 1.47x speed- ups compared to EdgeCPU, EdgeGPU, and vision transformer accelerator ViTCoD.
Objective: This study investigates the feasibility to apply transfer learning to develop compact, end-to-end algorithms optimized for the permanent, long-term operation of systems for the prediction of epileptic seizures. Methods: Thirteen artificial neural networks (ANNs), originally pretrained on ImageNet for image classification—including six compact models and seven milestone architectures—were repurposed for seizure prediction using two benchmark EEG datasets. The performance of these transferred models was evaluated against ANNs trained from scratch and leading state-of-the-art algorithms. Results: Transfer learning based models significantly outperformed models trained from scratch. Remarkably, lightweight models such as Shufflenet with only 1.4 million parameters and 50 million multiply-accumulate operations, achieved state-of-the-art performances. An input converter was introduced, expanding the effective receptive field and significantly enhancing prediction performance, particularly in compact models. Moreover, transfer learning algorithms performed effectively in challenging cases where EEG-specific ANNs and feature-based baselines failed, demonstrating their ability to capture features beyond domain-specific knowledge. Conclusion: Transfer learning is a promising strategy for developing compact and efficient algorithms well-suited for long-term seizure prediction. Significance: This research creates opportunities to advance seizure prediction technology beyond current methods.
This work presents a real-time 3D Gaussian Splatting-based SLAM (3DGS-SLAM) accelerator to support high-quality dense mapping in mobile environments. There are two main hardware challenges for 3DGS-SLAM acceleration; 1) Large power consumption due to non-uniform memory access and 2) Excessive a-computation caused by processing unnecessary pixels. To solve these issues, two key building blocks are integrated in the proposed 3DGS-SLAM accelerator. First, a Voxel-based Gaussian Management (VGM) divides the entire scene into multiple voxels enabling selective Gaussian fetching with streaming off-chip memory access. This method reduces external memory access by 26.2%. Second, ROI-based Pixel-wise Skipping (RPS) reduces 68.3 % of a-computation by skipping unnecessary pixels outside of ROI. As a result, the proposed 3DGS-SLAM accelerator is implemented on Intel's Cyclone V with a maximum clock frequency of 200 MHz. Evaluating on Replica dataset, it achieves 15.2 FPS 3DGS-SLAM enabling highquality dense mapping on mobile devices.
Obtaining data from neuromorphic sensors and processing it with Spiking Neural Networks (SNNs) is a promising solution to lower energy consumption in Artificial Intelligence systems. The current rarity of natively neuromorphic datasets promotes the development of software tools to translate input sensory data into spikes. However, highly bio-mimetic simulators can be challenging to implement on digital hardware. In this work, we evaluate the impact of audio-to-spikes encoding on an audio classification task regarding its accuracy, energy efficiency, and ease of hardware implementation. We compare a state-of-the-art, bio-mimetic cochlear encoder to a lighter and more hardware-friendly model, on the benchmark datasets Heidelberg Digits and Google Speech Commands. Our work shows that the latter model can reach higher accuracy and energy efficiency, paving the way for better neuromorphic systems.
Recently Transformer model has revolutionized the fields of natural language processing (NLP) and computer vision (CV). Even though it shows unprecedented efficiency and accuracy on most tasks, memory and computation overhead can be huge, making it inadequate for real-world applications such as edge devices. Post-training quantization (PTQ) can be an efficient way to overcome this bottleneck. In this paper, we propose a novel memory-aware mixed precision PTQ which will consider memory consumption of blocks and sub-blocks inside transformer model. Our motivation originates from profiling result which varies according to application. We evaluate the proposed methodology by employing both for language model for translation tasks and a vision-to-language model for image captioning tasks. Through our approach, we achieve significant memory savings while experiencing small degradation in accuracy. We are able to achieve 10x memory savings for the weights of the annotated transformer, 5x for T5 text-to-text, and 5.5x for weights of vision-to-language model.
This work presents an instruction-controlled analog In-Memory Computing (IMC) core to compute dot-product kernels of varying lengths and precisions parallelly, while achieving an energy efficiency of 107 TOPS/W at 8b precision. This is achieved by a combination of configurable multi-bit multiply-accumulate cells, low energy SAR ADCs with input gain control, and a novel kernel output routing approach. Using this IMC core, a full-fledged multi-core inference accelerator ASIC has been developed that achieves the target accuracy for a Voice Activity Detection usecase over PVT variations.
With the increasing demands of AI applications, especially in large-scale data processing and real-time decision-making, TCAM plays a crucial role in accelerating AI computations by enabling fast, parallel search operations. Our novel nonvolatile ternary content-addressable memory Nonvolatile-TCAM cell is well-suited to support the growing need for efficient, high-speed memory solutions in AI-driven systems, where both low power and rapid access times are critical. Recent developments in magnetic tunnel junctions (MTJs) based Nonvolatile-TCAMcells have aimed to achieve zero standby power. However, high search energy and reliability issues remain significant obstacles. To tackle these issues, we propose an 8T-2MTJ Nonvolatile-TCAM cell. This design leverages MTJs with complementary states and incorporates positive feedback from cross-coupled inverters, significantly improving search reliability. In the crucial path between matchline (ML) and ground, the suggested design only makes use of one transistor, thereby minimizing switching delay and enabling faster search operations. Our design outperforms existing Nonvolatile-TCAM cells, achieving a search delay of 0.0744 ns and reducing search energy consumption to 0.302 fJ/bit/search, prioritizing low energy usage and minimal search delay.
The complexity of RF PA behavior in 5G communication systems is driven by nonlinearities and long memory effects under wideband, high dynamic range signals; this poses significant challenges for existing models. Traditional Volterrabased models, such as the Generalized Memory Polynomial (GMP), struggle with overfitting, while neural network-based approaches require large datasets and exhibit high computational complexity both for training and inference. This paper presents a novel Piecewise Nonlinear AutoRegressive with eXogenous inputs (PW-NARX) model that combines the strengths of piecewise modeling and the NARX architecture to capture both nonlinear and memory effects efficiently over high dynamic ranges. Each sub-model in the piecewise framework operates within a different region of the input space, significantly reducing model complexity while maintaining high accuracy. Simulation results demonstrate that the PW-NARX model outperforms state-of-the-art models, achieving the lowest normalized mean square error (NMSE) of -39.18 dB and similar or better NMSE performance as other state-of-the-art models with fewer parameters.
Large Language Models (LLMs) have demonstrated remarkable capabilities across various domain tasks. Following this trend, this paper is dedicated to a new LLM-based design methodology for analog/mixed-signal (AMS) circuit design automation. The proposed LLM-based multi-agent circuit framework is capable of generating circuits based on given requirements. The framework leverages circuit-oriented retrieval-augmented generation (RAG) technology to retrieve specific circuit design knowledge and employs multi-agent collaboration. The proposed framework has been validated by designing fundamental AMS circuits, such as operational amplifiers (OPAMPs) and analog-to-digital converters (ADCs), which enhances the efficiency of AMS circuit design and promotes the development of AI-driven circuit design methodologies.
In the quest for energy-efficient neural network architectures, Spiking Neural Networks (SNNs) have emerged as a promising alternative to traditional Artificial Neural Networks (ANNs). However, the inherent SNN training complexity and the high spiking activity often limit their practical application. This paper explores hybrid neural networks (HNNs) that integrate ANN and SNN layers to optimize the trade-off between performance and energy efficiency. Our study systematically explores nearly exhaustive configurations of ANNs and SNNs on two datasets (CIFAR-10 and GSC V2) and five architectures (VGG16, VGG11, ResNet18, ResNet8, and a 4-layer CNN). Depending on the proportion of SNN and ANN layers, our hybrid models can reduce up to 2.45 times the energy consumption while maintaining the same ANN baseline accuracy, and even exceeding it for some models. These results highlight the potential of hybrid models in the development of energy-efficient neural networks and paves the way for future hardware implementations.
This study proposes a novel technique to improve the efficiency of the inference based on binary neural networks (BNNs). In the proposed technique, the threshold, used to binarize feature elements, is adaptively scaled by considering the spatial localities inherent in feature maps, to skip redundant operations without degrading inference accuracy. A BNN inference processor supporting the proposed technique is designed and implemented, and the effectiveness of the proposed technique is evaluated with the implementation results of the processor. The proposed technique reduces the overall latency by 17.0%, while the inference accuracy is degraded by less than 1.56% for the CIFAR10 classification task. The resource efficiency is enhanced by up to 36.7%.
Recent development in neuromorphic hardware focused on exploiting the temporal and spatial sparsity of Spiking Neural Networks (SNNs). Event driven accelerators have the benefit of lower latency and power compared to time-step driven architectures, but they are challenging to implement due to increased complexity of the neuron dynamics and irregular memory accesses. Here, we propose a scalable, flexible and customizable event-driven SNN accelerator designed for low latency and high capacity, with a hierarchical memory system and configurable neuron dynamics. The stateful nature of SNNs lead to frequent memory accesses, thus require a very high bandwidth to achieve a lower latency. While classic systolic array driven architectures opt for Static Random Access Memorys (SRAMs), SRAMs occupy a larger amount of area, limiting their capacity both on Application Specific ICs (ASICs) and Field Programmable Gate Arrays (FPGAs). In contrast, our accelerator leverages High Bandwidth Memory (HBM) as the main memory to achieve significantly higher capacity without sacrificing throughput. We benchmarked our accelerator on NMNIST and Spiking Heidelberg Digit (SHD) datasets. The system achieved 97.20% and 77.67% accuracy, respectively. Compared with state of the art implementations, our accelerator is able to process spike trains with very high time resolution, number of time-steps and little loss of accuracy compared to software baseline.
This paper presents a novel, low-latency, and power-efficient peripheral nervous system (PNS) on-chip classifier for real-time closed-loop adaptive neuromodulation. We propose an end-to-end closed-loop modulation system with a novel compute-in-memory (CiM) architecture based on magnetoresistive random-access memory (MRAM), enabling on-chip classification of neural signals. Using a custom deep neural network (DNN) and an end-to-end kernel fusion technique, the interface achieves minimal latency and power consumption, eliminating the need for external processors. This system is implemented in a 28-nm FD-SOI process, achieving 94.76% accuracy and <16 mu s/classification latency. This technology enables next-generation neuroprosthetic and therapeutic devices by providing a robust solution for real-time in-situ PNS classification.
Traffic congestion and road safety remain critical challenges in urban environments, driving the need for more effective traffic monitoring solutions. While recent advancements in computer vision have enhanced traffic perception, the dynamic viewpoint of autonomous vehicles is often insufficient for comprehensive traffic management. To address this gap, we propose an AI-driven framework for enhanced traffic scene understanding using static LiDAR sensors at road intersections. The system collects 3D point clouds from roadside static LiDAR sensors, providing a complete view of vehicles and pedestrians. We integrate state-of-the-art 3D object detection (i.e., PV-RCNN) and instance segmentation models (i.e., PointGroup3heads) to accurately identify road users and their locations. To validate the approach, we developed a 3D simulated traffic environment and generated a labeled dataset of diverse traffic scenarios. Our experimental results demonstrate the effectiveness of combining static LiDAR sensors with deep learning models for accurate scene understanding, offering a scalable solution for AI-driven traffic monitoring in complex urban areas.
DNA sequencing technology has revolutionized precision medicine by enabling personalized treatments, the early detection and prevention of diseases. However, existing methods for generating DNA analysis reports face challenges related to precision, completeness, and automation. Recent advances in large language models (LLMs) have substantially impacted modern biology and medicine. In this paper, we introduce a novel pipeline called ChatDNA, which uses a large medical language model based on vector databases and prompt learning. This approach facilitates the end-to-end generation of DNA sequencing data analysis reports, achieving an accuracy rate of 71.64% without quantization and 68.31% with INT8 quantization. Furthermore, the deployment of the quantized model on an integrated device equipped with a DNA sequencer allows for real-time, on-site DNA analysis, enhancing both the accessibility and efficiency of precision medicine. ChatDNA significantly improves the precision, depth, and automation of report generation, laying a solid foundation for AI-driven personalized precision medicine.
Deep Neural Network (DNN) inference workloads, such as object recognition in autonomous systems, place significant demands on performance and energy efficiency. Additionally, real-world DNN inference workloads are usually influenced by external factors, such as bad weather, which can lead to a degradation in accuracy. To address performance and power issues, techniques such as quantization and model pruning have been introduced, but these can worsen model robustness. This study explores mixed-precision DNN inference on a heterogeneous System-on-Chip (SoC) to balance performance and power consumption while maintaining model robustness. We allocate each DNN layer to either a performance- or energy-efficient accelerator, utilizing different computational precisions. In our work, a performance cost model is formutaled first to predict the total execution time. Then, we determine the schedules using reinforcement learning, with the cost model providing feedbacks. Our approach is evaluated on the NVIDIA Xavier SoC with commonly used DNN models, achieving 97.76% of full-precision accuracy, while reducing inference latency and energy consumption by up to 16% and 33%, respectively, compared to the state-of-the-art.
Combinatorial optimization widely exists in science and engineering problems. Some problems are NP-hard and thus become difficult to solve on edge devices due to limited power and computing resources. Quadratic Unconstrained Binary Optimization (QUBO) problem is an emerging model to approach numerous combinatorial problems. In this paper, we explore the potential of accelerating the solution of the QUBO problem using spintronic resonator arrays in combination with a neuromorphic swarm of spiking neural networks (SNN). This approach leverages the collaboration of multiple SNNs, performing a collaborative metaheuristic search that contains matrix and vector multiplication facilitated by the multiplication and accumulation (MAC) primitive based on spintronic resonator arrays. Our simulations demonstrate that the proposed method is promising in designing high-speed and energy-efficient solvers for various combinatorial optimization problems.
Dynamic token pruning has played an increasingly important role in reducing computational complexity and memory demands for the multi-head attention mechanism in Transformer-Based models. However, coarse-grained sparsification in existing methods overlooks contextual nuances, limiting hardware efficiency and optimization of model accuracy, complexity, and energy consumption. To address these limitations, we propose a head-wise adaptive sparsification scheme that selectively removes less relevant tokens for each attention head. Such finer-grained, context-aware pruning, with an adaptive pruning rate, improves inference latency while effectively maintaining model accuracy. Moreover, the sparse feed-forward networks can be mapped to head-wise dense matrix multiplications, further enhancing hardware utilization and reducing energy consumption. Experimental results on the BERT-Base model demonstrate that our approach achieves an average of 57.43% reduction in energy-delay product compared to the unpruned baseline, outperforming the state-of-the-art by 22.34%.
