Fully Homomorphic Encryption (FHE) enables encrypted data processing on untrusted cloud servers, crucial for privacy-sensitive applications. Despite its potential, performance overheads (about $10,000\times$10,000x slower) limit adoption. ASIC accelerators outperform GPUs/FPGAs by optimizing specific operations but rely on costly 7nm processes and large on-chip memory, hindering cost-effective deployment. Balancing efficiency with manufacturing constraints remains critical. This paper presents TensorFHE+, a GPU-optimized FHE acceleration framework leveraging Tensor Cores to accelerate Number Theoretic Transform (NTT) operations. Key innovations include: 1) Decomposing CKKS kernels into vector/matrix operations for hardware utilization; 2) Vectorized modulo arithmetic; 3) Data layout optimization for memory efficiency. Evaluated on NVIDIA A100, TensorFHE+ outperforms TensorFHE[1] by 1.44 x in average (up to 1.69x on ResNet-20) and surpasses prior GPU implementations [2], [3]. The design also demonstrates compatibility with commercial linear algebra accelerators, enabling efficient FHE deployment.

3D Gaussian Splatting (3DGS) is a breakthrough in 3D reconstruction using 3D Gaussians. However, even on high-end GPUs like the NVIDIA A100, reconstructing complex scenes remains time-consuming, taking over 15 minutes. The main bottleneck is a-computation, which accounts for 71.25% of training workload, yet 93.03% of it is invalid due to the localized influence of Gaussians. To address this issue, we propose Cambricon-GS, an accelerator for 3DGS training with Gaussian-Pixel hybrid parallelism. At the software level, we introduce a hybrid parallel workflow that breaks the limitation of conventional pixel-only parallelism through two key techniques: Center-Pixel Gaussian Culling (CPGC), which eliminates invalid Gaussians early, and Seed-Driven Gaussian Region Exploration (SDGRE), which reduces invalid computation for partially valid Gaussians by selectively exploring valid regions. Overall, the workflow significantly reduces a computations, lowering the workload to 17.99%. At the hardware level, Cambricon-GS decouples alpha-computation and alpha blending into GUnits and PUnits, organized in a 2D mesh-based NoC that supports asynchronous execution and efficient data routing. We further boost performance via Gaussian/Pixel load balancing and tiled SSIM based pipelining. The evaluation results show that Cambricon-GS achieves 19.63x, 14.86x, 15.42x, 2.98x and 2.63x speedup, and 78.62x, 63.00x, 61.72x, 3.89x and 3.22x energy saving, compared to A100, GSCore, GBU, GSArch, and GauSPU, respectively, with negligible image quality loss.

Quantization has become a critical technique for enhancing the efficiency of deep neural network (DNN) training, especially for large-scale models. Microscaling (MX) format is a recently emerged and widely adopted standardized data format for quantized training. Microscaling format leverages shared scaling factors to reduce the storage and computational overhead associated with quantized training. However, despite its potential, the Microscaling training algorithm faces significant performance challenges when deployed on existing hardware architectures, primarily due to the lack of effective support for on-the-fly quantization and the high communication overhead of weight updates. These limitations prevent efficient execution of Microscaling on conventional DNN accelerators, such as GPUs and TPUs. To address these challenges, we propose Cambricon-QM, a novel hybrid architecture specifically designed to support Microscaling format training. Cambricon-QM integrates an ASIC acceleration core with a near-data processing (NDP) engine, allowing it to efficiently handle the unique demands of on-the-fly quantization and high-precision weight updates. Cambricon-QM minimizes memory access overhead and supports quantized computation in the Microscaling format, resulting in substantial improvements in energy efficiency and performance over existing systems. Experimental results demonstrate that Cambricon-QM achieves up to 14.56× energy efficiency and 8.54× performance improvements compared to conventional GPUs.

Unary computing, whose core arithmetics (e.g., multiplication) require only simple logic gates, has enabled efficient DNN processing, especially on strictly power-constrained devices. However, unary computing still confronts the power efficiency bottleneck for buffering unary bitstreams. The buffering of unary bitstreams requires accumulating bits into large bitwidth binary numbers. The large bitwidth binary number needs to activate all bits per cycle in case of carry propagation. As a result, the accumulation process accounts for 32%-70% of the power budget. To push the boundary of power efficiency, we propose Cambricon-U, a systolic random increment memory architecture featuring efficient accumulation. By leveraging skew number data format, Cambricon-U only activates no more than three bits (instead of all bits) from each number per accumulating cycle. Experimental results show that Cambricon-U reduces 51% power on unary accumulation, and improves 1.18-1.45× energy efficiency over uSystolic, the SOTA unary computing scheme baseline, with -1.9%∼+0.77% area overhead. Further, we propose Cambricon-UX, which exploits bit-level sparsity in unary computing to save energy. Compared to Cambricon-U, Cambricon-UX achieves 1.28-2.54× energy efficiency.

Diffusion transformers (DiTs) have demonstrated remarkable success in text-to-video generation. However, the self-attention mechanism in DiTs imposes significant computational and memory burdens, particularly when handling long patch sequences like high-resolution or long-time videos. While sparse attention shows promise in reducing self-attention costs, existing approaches struggle to deliver performance gains due to the unique challenges in DiTs, i.e., varied sparse patterns across layers and timesteps, and the cumulative nature of inference errors over timesteps. In this article, we propose DASA, an algorithm-hardware co-design that effectively addresses these challenges of attention sparsification in DiTs. Specifically, leveraging the insight that the generation quality is primarily influenced by overall distribution drift rather than changes in specific values, we introduce a novel distribution-aware filtering (DAF) mechanism for sparsification. To further accelerate the process, we design a specialized filtering unit that enables fast candidate selection based on the proposed DAF mechanism. Experimental results show that DASA achieves 2.52 & times; speed up compared to A100 GPU, and up to1.22 & times; speedup over state-of-the-art accelerators for self-attention computation.

Neural Scene Representation (NSR) is a promising technique for representing real scenes. By learning from dozens of 2D photos captured from different viewpoints, NSR computes the 3D representation of real scenes. However, the performance of NSR processing running on GPU is insufficient for applications. Cambricon-R achieves high performance of more than 60 scenes per second, but at the cost of modeling quality. In this paper, we propose Cambricon-SR, an algorithm-hardware co-designed accelerator to improve NSR performance and quality. First, we propose a novel NSR algorithm with sparse encoding table, it achieves more than 80% of sparsity for the encoding table with negligible accuracy loss. Second, to efficiently eliminate the invalid memory access requests to the pruned entries, we propose the sparse index unit based on sequential SRAM access to address the challenge of irregular memory access to the sparse structure bitmap. It achieves a throughput increase of 7.54x for the encoding stage with an additional area overhead of 8.59%. Third, we propose the dynamic shared buffer for the MLP units to reduce the buffer usage by 85.3% so that we can increase the number of the MLP units. We conduct experimental evaluations on 8 typical scenes. The results demonstrate that, compared with A100 GPU and Cambricon-R, Cambricon-SR achieves 1259x and 4.12x speedup, and reduces the energy consumption by 1139x and 2.98x for each training iteration of NSR, respectively.

Dynamic Graph Neural Networks (DGNNs) have demonstrated significant potential in handling temporal graph-structured data. Typical DGNNs combine GNNs to process the structural information of graph and RNNs to capture the dynamic temporal information of graphs. However, current GNN accelerators process DGNNs by treating graph snapshots as static entities, leading to substantial redundant computations and memory accesses. While some DGNN accelerators have attempted to mitigate this redundancy, they face the challenge of maintaining accuracy for incremental processing due to the nonlinear activation function in GNNs. To address these issues, we propose a novel nonlinear isolation mechanism for incremental processing of DGNNs, which eliminates all redundant operations without accuracy loss. Additionally, we introduce a vertex-wise computation scheme for RNNs to minimize redundancy and reduce off-chip memory access. To validate the effectiveness of the mechanism, we implemented software optimizations on GPUs, achieving an 8.53x performance improvement over full-graph computation schemes. To address the inefficiency of graph topology operations on GPUs, we propose the Cambricon-DG accelerator, which features a dedicated hardware pipeline, an efficient topology management engine for graph traversal and topology sorting, and a hybrid computation engine that supports both GNN and RNN computations, thus improving hardware utilization. We evaluate Cambricon-DG on seven real-world dynamic graph datasets using five representative DGNN models. The results show that Cambricon-DG achieves an average speedup of 49.73x, 12.15x, and 7.03x, and an average energy saving of 41.31x, 9.47x, and 5.23x over the state-of-the-art static/dynamic GNN accelerators I-GCN, RACE, and DeltaGNN, respectively.

Deep learning has achieved great success in numerous application areas at the cost of high computational complexity. To meet the ever-increasing computational demand, commodity hardware platforms (e.g., CPUs and GPUs) offer abundant computing resources including scalar, vector, and tensor units for deep learning that could execute in parallel. However, existing top-down tiling-based deep learning compilers often generate a homogeneous mapping from the given tensor computation task to hardware arithmetic instructions, failing to utilize different computing units simultaneously to achieve higher performance. In this paper, we propose Mosaic, a bottom-up tessellation-based deep learning compiler that directly tessellates the given tensor computation task with varying instructions, forming a heterogeneous instruction-to-task mapping to exploit instruction-level parallelism (ILP) across different computing units. The key that enables such tessellation is the iTex abstraction, which models the relationship between the instruction operations and its semantics with formalized affine functions. Based on the iTex, we propose a heuristic approach to efficiently generate various tessellation plans. Further, we propose the iTex scheduling technique to orchestrate the execution of instructions, reducing potential structural hazards and maximizing the exploitable ILP. Our extensive evaluation shows that Mosaic achieves an average speedup ranging from 1.08x to 1.28x across multiple hardware platforms compared to highly optimized vendor libraries. Mosaic also achieves an average speedup of 1.34x over the best existing baselines on real-world operators extracted from LLMs. More importantly, Mosaic reaches up to 106% of the GPU Tensor Core theoretical peak throughput, demonstrating its effective exploitation of ILP.

Symbolic regression (SR), the process of formulating a mathematical expression based on observed data points, is a fundamental task in artificial intelligence but is often hindered by its intense computational demands. Deep-learning-based SR methods (DSR) aim to alleviate these demands by breaking down the SR process into two stages: 1) neural network (NN) inference and 2) Broyden-Fletcher-Goldfarb-Shanno (BFGS) optimization. Although NN accelerators can expedite the NN stage, the performance of the BFGS optimization is compromised due to its poor performance for the variety of transcendental functions. Moreover, the distinct computational characteristics of NN inference and BFGS cause not only low hardware utilization but also significant area waste. To address these issues, we propose Harmonia, a unified architecture with the neural transcendental function unit (NTFU) and the Unified Array for efficient DSR. The NTFU utilizes the radial basis function network (RBFN) as a universal approximator for various transcendental functions, which significantly reduces the heavy transcendental function computation cost. We further propose an efficient training algorithm called random nonlinear optimization (RNO) to obtain a lightweight RBFN without accuracy loss. Moreover, Harmonia supports configurable dataflow which integrates the two computing stages into the Unified Array. Experimental results show that Harmonia achieves hardware utilization of 83.83%, on average. Compared to the GPU baseline, Harmonia achieves 4.8x speedup and 47.6x energy saving, alongside considerable low area cost.

Diffusion models have made significant progress in current image generation tasks, thus becoming a prominent area of research. Diffusion models necessitate repetitive iterations on minimally altered input data across timesteps, each timestep requiring the recalculation of the entire model, resulting in a remarkable computational redundancy and substantial hardware expenditures. Performing differential computing on input data seems to be a feasible approach for addressing such computational redundancy and improving hardware efficacy. However, non-linear operations (particularly activation functions) necessitate the merging of deltas (i.e., differential values) with raw inputs repeatedly to ensure computational correctness, leading to significant memory access for loading raw inputs, which fragmentedly blocks the forwarding of deltas throughout the network and undermines performance. To solve this problem, we propose Cambricon-D, a full-network differential computing architecture with concise memory access. While maintaining the computational efficiency brought by differential computing, Cambricon-D employs a sign-mask dataflow, which requires only the loading of 1-bit signs (instead of large bitwidth raw inputs), thereby facilitating the seamless forwarding of deltas and effectively mitigating memory access overheads. Experimental results show that, compared to Diffy, Cambricon-D's dataflow reduces 66% similar to 82% off-chip memory access. In total, Cambricon-D achieves 1.46x similar to 2.38x speedup over A100 on various diffusion models with different resolutions.

Domain adaptive object detection (DAOD) aims to generalize detectors trained on an annotated source domain to an unlabelled target domain. As the visual-language models (VLMs) can provide essential general knowledge on unseen images, freezing the visual encoder and inserting a domain-agnostic adapter can learn domain-invariant knowledge for DAOD. However, the domain-agnostic adapter is inevitably biased to the source domain. It discards some beneficial knowledge discriminative on the unlabelled domain, \ie domain-specific knowledge of the target domain. To solve the issue, we propose a novel Domain-Aware Adapter (DA-Ada) tailored for the DAOD task. The key point is exploiting domain-specific knowledge between the essential general knowledge and domain-invariant knowledge. DA-Ada consists of the Domain-Invariant Adapter (DIA) for learning domain-invariant knowledge and the Domain-Specific Adapter (DSA) for injecting the domain-specific knowledge from the information discarded by the visual encoder. Comprehensive experiments over multiple DAOD tasks show that DA-Ada can efficiently infer a domain-aware visual encoder for boosting domain adaptive object detection. Our code is available at https://github.com/Therock90421/DA-Ada.

Domain adaptive object detection (DAOD) aims to infer a robust detector on the target domain with the labelled source datasets. Recent studies utilize a feature extractor shared on the source and target domains to capture the domain-invariant features and the task-relevant information with both feature-alignment constraint and source annotations. However, the feature extractor shared across domains discards partial task-relevant information of the target domain due to the domain gap and lack of target annotations, leading to compromised discrimination capabilities within target domain. To this end, we propose a novel REmainder Adaptive CompensaTion network (REACT) to adaptively compensate the extracted features with the remainder features for generating task-relevant features. The key insight is that the remainder features contain the discarded task-relevant information, so they can be adapted to compensate for the inadequate target features. Especially, REACT introduces an additional remainder branch to regain the remainder features, and then adaptively utilizes them to compensate for the discarded task-relevant information, improving discrimination on the target domain. Extensive experiments over multiple cross-domain adaptation tasks with three baselines demonstrate that our approach gains significant improvements and achieves superior performance compared with highly-optimized state-of-the-art methods.

Research on emergent communication has recently gained significant traction as a promising avenue for the linguistic community to unravel human language's origins and explore artificial intelligence's generalization capabilities. Current research has predominantly concentrated on recognizing qualitative patterns of object attributes(e.g., shape and color) and paid little attention to the quantitative relationship among object quantities which is known as the part of numerical concepts. The ability to generalize numerical concepts, i.e., counting and calculations with unseen quantities, is essential, as it mirrors humans' foundational abstract reasoning abilities. In this work, we introduce the NumGame, leveraging the referential game framework, forcing agents to communicate and generalize the numerical concepts effectively. Inspired by the human learning process of numbers, we present a two-stage training approach that sequentially fosters a rudimentary numerical sense followed by the ability of arithmetic calculation, ultimately aiding agents in generating semantically stable and unambiguous language for numerical concepts. The experimental results indicate the impressive generalization capabilities to unseen quantities and regularity of the language emergence from communication.

Heterogeneous collaborative computing with NPU and CPU has received widespread attention due to its substantial performance benefits. To ensure data confidentiality and integrity during computing, Trusted Execution Environments (TEE) is considered a promising solution because of its comparatively lower overhead. However, existing heterogeneous TEE designs are inefficient for collaborative computing due to fine and different memory granularities between CPU and NPU. 1) The cacheline granularity of CPU TEE intensifies memory pressure due to its extra memory access, and 2) the cacheline granularity MAC of NPU escalates the pressure on the limited memory storage. 3) Data transfer across heterogeneous enclaves relies on the transit of non-secure regions, resulting in cumbersome re-encryption and scheduling. To address these issues, we propose TensorTEE, a unified tensor-granularity heterogeneous TEE for efficient secure collaborative tensor computing. First, we virtually support tensor granularity in CPU TEE to eliminate the off-chip metadata access by detecting and maintaining tensor structures on-chip. Second, we propose tensor-granularity MAC management with predictive execution to avoid computational stalls while eliminating off-chip MAC storage and access. Moreover, based on the unified granularity, we enable direct data transfer without re-encryption and scheduling dilemmas. Our evaluation is built on enhanced Gem5 and a cycle-accurate NPU simulator. The results show that TensorTEE improves the performance of Large Language Model (LLM) training workloads by 4.0x compared to existing work and incurs only 2.1% overhead compared to non-secure training, offering a practical security assurance for LLM training.

Deep learning trends to use low precision numeral formats to cope with the ever-growing model sizes. For example, the large language model LLaMA2 has been widely deployed in 4-bit precision. With larger models and fewer unique values caused by low precision, an increasing proportion of arithmetic in matrix multiplication is repeating. Although discussed in prior works, such value redundancy has not been fully exploited, and the cost to leverage the value redundancy often offsets any advantages. In this paper, we propose to primitivize the matrix multiplication, that is decomposing it down to the 1-ary successor function (a.k.a. counting) to merge repeating arithmetic. We revisited various techniques to propose Cambricon-C SA, a 4-bit primitive matrix multiplication unit that doubles the energy efficiency over conventional systolic arrays. Experimental results show that Cambricon-C SA can achieve 1.95x energy efficiency improvement compared with MAC-based systolic array.

With the rapid development of Artificial Intelligence of Things (AIoT), customizing and optimizing operating system (OS) kernel configurations for various AIoT application scenarios is crucial for maximizing system performance. However, existing approaches falter due to the overwhelming problem complexity (i.e., over 15,000 configuration options in the Linux kernel), together with the huge evaluation costs and error-prone options that may result in OS boot-up failure, which all make it an unresolved problem to optimize the Linux kernel automatically. In this paper, we introduce AutoOS, a novel framework exploiting Large Language Models for customizing and optimizing OS kernel configurations automatically for various AIoT application scenarios. Inspired by the inherently directory-structured kernel configuration process, we first formulate our research problem as optimizing on a dynamic tree. We then propose a novel framework integrating a state machine-based traversal algorithm as the observe-prune-propose-act-correct loop, which can effectively refine the optimization space and ensure a successful OS boot-up. Experimental results show that AutoOS can automatically customize and optimize the OS kernel configurations without human effort. More importantly, AutoOS even achieves better performance by up to 25% than vendor-provided configuration.

In the current landscape, high-resolution (HR) videos have gained immense popularity, promising an elevated viewing experience. Recent research has demonstrated that the video super-resolution (SR) algorithm, empowered by deep neural networks (DNNs), can substantially enhance the quality of HR videos by processing low-resolution (LR) frames. However, the existing DNN models demand significant computational resources, posing challenges for the deployment of SR algorithms on client devices. While numerous accelerators have proposed solutions, their primary focus remains on client-side optimization. In contrast, our research recognizes that the HR video is originally stored in the cloud server and presents an untapped opportunity for achieving both high accuracy and performance improvements. Building on this insight, this paper introduces an end-to-end video CODEC-assisted super-resolution (E2SR+) algorithm, which tightly integrates the cloud server with the client device to deliver a seamless and real-time video viewing experience. We propose the motion vector search algorithm executed in the cloud server, which can search the motion vectors and residuals for part of HR video frames and then pack them as addons. We also design an auto-encoder algorithm to down-sample the residuals to save the bitstream cost while guaranteeing the quality of the residuals. Lastly, we propose the reconstruction algorithm performed in the client to fast reconstruct the corresponding HR frames using the addons to skip part of DNN computations. To implement the E2SR+ algorithm, we design the corresponding E2SR+ architecture in the client, which achieves significant speedup with minimal hardware overhead. Given that the environmental condition varies in the server-client hierarchies, we believe that simply applying E2SR+ to all frames is irrational. Accordingly, we offer an environmental condition aware system to chase the best performance while adapting to the diverse environment. In the system, we design a linear programming (LP) model to simulate the environment and allocate frames to three existing mechanisms. Our experimental results demonstrate that the E2SR+ algorithm enhances the PSNR by 1.2, 2.5, and 2.3 compared to the SOTA methods “EDVR”, “BasicVSR”, and “BasicVSR++”, respectively. In terms of performance, the E2SR+ architecture offers significant improvements over existing SOTA methods. For instance, while BasicVSR++ requires 98ms on Nvidia V100 GPU to generate a 1280 × 720 HR frame, the E2SR+ architecture reduces the execution time to just 39ms, highlighting the efficiency and effectiveness of our proposed method. Overall, the E2SR+ architecture respectively achieves 1.4 ×, 2.2 ×, 4.6 ×, and 442.0 × performance improvement compared to ADAS, ISRAcc, NVIDIA V100 GPU, and CPU. Lastly, the proposed system showcases its superiority and surpasses all the existing mechanisms in terms of execution time when varying environmental conditions.

Deploying advanced large language models on edge devices, such as smartphones and robotics, is a growing trend that enhances user data privacy and network connectivity resilience while preserving intelligent capabilities. However, such a task exhibits single-batch computing with incredibly low arithmetic intensity, which poses the significant challenges of huge memory footprint and bandwidth demands on limited edge resources. To address these issues, we introduce Cambricon-LLM, a chiplet-based hybrid architecture with NPU and a dedicated NAND flash chip to enable efficient on-device inference of 70B LLMs. Such a hybrid architecture utilizes both the high computing capability of NPU and the data capacity of the NAND flash chip, with the proposed hardware-tiling strategy that minimizes the data movement overhead between NPU and NAND flash chip. Specifically, the NAND flash chip, enhanced by our innovative in-flash computing and on-die ECC techniques, excels at performing precise lightweight on-die processing. Simultaneously, the NPU collaborates with the flash chip for matrix operations and handles special function computations beyond the flash's on-die processing capabilities. Overall, Cambricon-LLM enables the on-device inference of 70B LLMs at a speed of 3.44 token/s, and 7B LLMs at a speed of 36.34 token/s, which is over 22X to 45X faster than existing flash-offloading technologies, showing the potentiality of deploying powerful LLMs in edge devices.

Tensor program optimization (TPO) based on pre-trained models can effectively reduce the computing time of deep neural networks. However, training of such models is prohibitively expensive, which highly depends on a large-scale dataset and thus requires tremendous time-consuming performance measurements (more than 1 million) on target platforms. In this paper, we propose BALTO, a fast TPO approach with biased-diversity-based active learning, aiming at reducing much lower training costs under similar optimization accuracy.The key insight is that random sampling of existing approaches suffers from a heavy redundancy of low-performance programs, which incurs tremendous duplicated time-consuming measurements. Inspired by this, BALTO removes such redundancy by introducing active learning (AL) to TPO for a much lower training cost. However, applying AL with a brute-force way in BALTO can lead to an overestimation problem. To address this, we further propose a biased-diversity-based diversity scheme specially designed for BALTO. We compare BALTO against TenSet on $6$ typical hardware platforms over $2$ learning models. Experimental results show that, on average, BALTO only requires 5% of the total performance measurements of TenSet to achieve the same or higher model accuracy. Moreover, the optimized tensor programs even outperform that of TenSet by 1.06% due to higher model accuracy.

With the increasing demand of computational power in artificial intelligence(AI)algorithms,dedicated accelerators have become a necessity.However,the complexity of hardware architectures,vast design search space,and complex tasks of accelerators have posed significant challenges.Tra-ditional search methods can become prohibitively slow if the search space continues to be expanded.A design space exploration(DSE)method is proposed based on transfer learning,which reduces the time for repeated training and uses multi-task models for different tasks on the same processor.The proposed method accurately predicts the latency and energy consumption associated with neural net-work accelerator design parameters,enabling faster identification of optimal outcomes compared with traditional methods.And compared with other DSE methods by using multilayer perceptron(MLP),the required training time is shorter.Comparative experiments with other methods demonstrate that the proposed method improves the efficiency of DSE without compromising the accuracy of the re-sults.