Large Language Models (LLMs) have attracted extensive attetion for their remarkable performance across a variety of tasks. However, the considerable computational and memory demands associated with LLMs present challenges for ensuring inference speed performance when deployed in resource-constrained scenarios. Current researches widely adopt model compression techniques and have greatly reduced the computational and memory overhead. Meanwhile, some frameworks that serve as deployment tools for large models can provide optimized deployment strategies based on the characteristics of hardware architectures. In this work, we explored LLM optimization performance on the ARMv9 computing platform, which is mainstream for intelligent edge usage. The inference efficiency was enhanced and memory usage was reduced by lowering dequantization cost, rewriting specific functions in LLaMA.cpp framework and using advanced compilation optimization techniques. Through optimizating and evaluating of the Qwen-1.8B model on ARMv9, inference performance improvement of 23.36x was achieved in decoding stage and 1.68 x in prefilling stage over the baseline with only a 1.04% loss in accuracy. All the developed code in this work is open source at https://github.com/CEATRG/ LLaMA.cpp- arm.
Phase-Change Memory (PCM) based resistive crossbars are a promising technology for energy-efficient neural network inference in analog in-memory computing (AIMC) systems. However, a conductance drift in PCM devices degrades performance over time, posing a challenge for applications requiring long-term stability. In order to overcome such problems, we propose a drift-aware regularization framework that stabilizes weight drift during training, complemented by an attention-based mechanism to prioritize critical features. Our approach significantly improves upon traditional hardware-aware (HWA) training and achieves more robust inference performance in the presence of conductance drift. In a five-year drift simulation, our method reduces classification error to 3.75%, compared to 9.14% with standard HWA training on a benchmark two-layer perceptron for the MNIST dataset. When combined with global drift compensation, the error is further reduced to 2.12%. These results demonstrate the effectiveness of our drift-aware regularization in enhancing the stability and accuracy of neural networks on AIMC hardware, offering a scalable, energy-efficient solution for inference in resource-constrained environments.
While vision transformers excel in various computer vision tasks, their high computational cost limits use on resource-constrained devices, highlighting the need for complexity reduction. In this paper, we present dynamic patch pruning for low complexities of vision transformers (DPP-ViT). To identify relatively more important patches, the column-wise accumulations of attention maps are computed and those are used as importance scores. Through DPP-ViT with block-wise importance score thresholds, our approach considers imagewise difficulties and block-wise sensitivities, removing the patches that contribute the least to accuracies. Additionally, we present a reconfigurable accelerator that dynamically changes dataflow and PE structure by applying pruning-aware row-level reconfiguration. DPP-ViT achieves 47% computation reduction with a minor -0.25% degradation on DeiT-B model in ImageNet top-1 accuracy. The proposed reconfigurable accelerator also achieves 47.96x/ 4.36x/ 1.47x speed- ups compared to EdgeCPU, EdgeGPU, and vision transformer accelerator ViTCoD.
Objective: This study investigates the feasibility to apply transfer learning to develop compact, end-to-end algorithms optimized for the permanent, long-term operation of systems for the prediction of epileptic seizures. Methods: Thirteen artificial neural networks (ANNs), originally pretrained on ImageNet for image classification—including six compact models and seven milestone architectures—were repurposed for seizure prediction using two benchmark EEG datasets. The performance of these transferred models was evaluated against ANNs trained from scratch and leading state-of-the-art algorithms. Results: Transfer learning based models significantly outperformed models trained from scratch. Remarkably, lightweight models such as Shufflenet with only 1.4 million parameters and 50 million multiply-accumulate operations, achieved state-of-the-art performances. An input converter was introduced, expanding the effective receptive field and significantly enhancing prediction performance, particularly in compact models. Moreover, transfer learning algorithms performed effectively in challenging cases where EEG-specific ANNs and feature-based baselines failed, demonstrating their ability to capture features beyond domain-specific knowledge. Conclusion: Transfer learning is a promising strategy for developing compact and efficient algorithms well-suited for long-term seizure prediction. Significance: This research creates opportunities to advance seizure prediction technology beyond current methods.
This work presents a real-time 3D Gaussian Splatting-based SLAM (3DGS-SLAM) accelerator to support high-quality dense mapping in mobile environments. There are two main hardware challenges for 3DGS-SLAM acceleration; 1) Large power consumption due to non-uniform memory access and 2) Excessive a-computation caused by processing unnecessary pixels. To solve these issues, two key building blocks are integrated in the proposed 3DGS-SLAM accelerator. First, a Voxel-based Gaussian Management (VGM) divides the entire scene into multiple voxels enabling selective Gaussian fetching with streaming off-chip memory access. This method reduces external memory access by 26.2%. Second, ROI-based Pixel-wise Skipping (RPS) reduces 68.3 % of a-computation by skipping unnecessary pixels outside of ROI. As a result, the proposed 3DGS-SLAM accelerator is implemented on Intel's Cyclone V with a maximum clock frequency of 200 MHz. Evaluating on Replica dataset, it achieves 15.2 FPS 3DGS-SLAM enabling highquality dense mapping on mobile devices.
