Generative AI (GenAI) is one of the most critical applications today, continually challenging the limits of semiconductor technology. We introduce a very fine-grained 3D memory-on-logic architecture along with a novel data mapping strategy to support Large Language Model (LLM)-based GenAI, including both pre-fill and generation stages. Our conceptual analysis shows how ultradense 3D connectivity can enhance text generation speed and energy-efficiency well-beyond current limits. Preliminary findings from a basic analytical model indicate that the single batch autoregressive generation rate for Llama 3.2 1B could surpass 5K tokens/sec by maximizing weight locality and enhancing memory bandwidth through massively parallel 3D links between Multiply-Accumulate (MAC) units in the logic tier and their dedicated memory partitions in the 3D stack. We also explore the impact of advanced logic nodes and quantify their benefits in reducing prefill latency. Finally, we examine the challenges associated with memory access power and power density under extreme bandwidth conditions and present pipelined access strategies to address them.

This article presents an static random-access memory (SRAM)-based digital compute-in-memory (CIM) compiler implemented with 3 nm high-kappa metal gate (HKMG) FinFET technology, supporting flexible INT8 and FP16 formats for weight and activation multiply-accumulate (MAC) operations, offering configuration flexibility, high accuracy, and improved area and power efficiency. The FP16 digital computing in memory (DCIM) compiler achieves high arithmetic accuracy by adopting a multiplication-first (MF) FP computation flow, while the INT8 DCIM compiler provides improved power-performance-area (PPA). The DCIM compiler engine enables the automatic generation of flexible CIM macro configurations based on the tiling of block-level leaf cells. Our test chip in 3 nm technology is fully validated and exhibits 124.6 TOPS/W at 0.5 V and 90.2 TOPS/mm(2) at 1.1 V for INT8 DCIM, and 28.6 TFLOPS/W at 0.5 V and 17.4 TFLOPS/mm(2) at 1.1 V for FP16 DCIM, respectively.

This paper presents a Compute-in-memory (CIM) compiler featuring multiple data format and multiple MAC units per weight, supporting flexible data format, better utilization and throughput. We implemented various configurations in a test chip with our 2 nm nanosheet technology node and test chip measurement results show $\mathrm{V}_{\text{MIN}}<0.38 \mathrm{V}, 234.4$ TOPS/W at 0.5 V and $511.9 \text{TOPS} / \text{mm}^{2}$ at 1.2 V.

Digital compute-in-memory (DCIM) has emerged as a promising technique to accelerate deep neural networks by minimizing data-movement and performing near/in-memory computation. While floating-point (FP) arithmetic being widely used in modern AI workloads due to its numerical robustness, supporting FP-DCIM brings design challenges, including dynamic range management, accumulation precision, FP arithmetic strategies and overheads. To address these challenges, we developed hardware-accurate convolutional and fully-connected layer operators to precisely emulate the multiply-and-add (MAC) operation of FP-DCIM. This enabled end-to-end accuracy evaluation under specific hardware design choices along with their quantified impact on both numerical behavior and hardware efficiency. We conduct a comprehensive algorithm-hardware co-design exploration across diverse AI workloads to identify the optimal FP-DCIM architecture that minimizes hardware cost under strict accuracy constraints. Specifically, we show that product mantissa can be truncated to 14b and 12b for FP16-DCIM and FP8-DCIM respectively with no accuracy degradation, while incorporating rounding allows even lower bit-width. These observations further enable exponent-dependent MAC-skipping as an orthogonal iso accurate strategy for additional energy savings. Hardware evaluation on 3nm FP16-DCIM macros indicates that combining the above strategies can reduce power by >37% without compromising accuracy performance.

To accelerate AI applications, numerous data formats and physical implementations of matrix multiplication have been proposed, creating a complex design space. This paper studies the efficient MAC implementation of the integer, floating-point, posit, and logarithmic number system (LNS) data formats and Microscaling (MX) and Vector-Scaled Quantization (VSQ) block data formats. We evaluate the area, power, and numerical accuracy (evaluated as signal-to-quantization noise ratio) of >25,000 MAC designs spanning each data format and several key design parameters such as the inner product size and accumulation width. We find that for the same numerical accuracy, pareto optimal MAC designs with emerging data formats (LNS16, MXINT8, VSQINT4) achieve 1.8x, 2.2x, and 1.9x TOPs/W improvement compared to FP16, FP8, and FP4 dot product implementations.

Compute-in-memory (CIM) is being widely explored to minimize power consumption related to data movement and multiply-and-accumulate (MAC) operations for AI edge devices. Compared to analog based CIMs, digital-based CIMs (DCIM), which include small, distributed SRAM banks and a customized MAC unit, realize massively parallel computation with no accuracy loss and better power-performance-area (PPA) scaling with advanced technologies. However, balancing operating efficiency per area (TOPS/mm 2 ) and bit density (Mb/mm 2 ) is one of the challenges in prior DCIMs because of low operating throughput caused by bit-serial input and a small number of rows. In this paper, we introduce a 3nm SRAM-based DCIM macro, which is based on foundry 6T SRAM bit cells and a parallel-MAC scheme to improve bit cell density and operating throughput. The DCIM macro is implemented with CIM BIST in a test chip, and confirms ultra-low voltage MAC operation down to 360mV, and 1.5GHz operation at 0.9V. The DCIM achieves 32.5TOPS/W (assuming a 25% input toggle rate and a 50% weight = 1 distribution), 55.0TOPS/mm 2 and 3.78 Mb/mm 2 .

In this work, we quantify the impact of power supply noise (PSN) in 3D-HI architectures on the errors in ADC and RRAM array outputs and optimize the 3D PDN and ADC designs to maximize inference accuracy in compute-in-memory (CIM) hardware. We propose a device-HI -application-level evaluation methodology to evaluate the impact of PDN design parameters on CIM inference accuracy. For our assumed 3D CIM hardware, an areal distribution of through-silicon vias (TSVs) and u-bumps, and a fine-tuned PSN-aware successive approximation register-ADC (SAR-ADC) achieves a 90% inference accuracy compared to 47% with a baseline 3D design at iso-power and iso-area. These insights can be useful for multi -die design convergence for edge intelligent CIM chips.

Augmented Reality and Virtual Reality have emerged as the next frontier of intelligent image sensors and computer systems. In these systems, 3D die stacking stands out as a compelling solution, enabling in situ processing capability of the sensory data for tasks such as image classification and object detection at low power, low latency, and a small form factor. These intelligent 3D CMOS Image Sensor (CIS) systems present a wide design space, encompassing multiple domains (e.g., computer vision algorithms, circuit design, system architecture, and semiconductor technology, including 3D stacking) that have not been explored in-depth so far. This article aims to fill this gap. We first present an analytical evaluation framework, STAR-3DSim, dedicated to rapid pre-RTL evaluation of 3D-CIS systems capturing the entire stack from the pixel layer to the on-sensor processor layer. With STAR-3DSim, we then propose several knobs for PPA (power, performance, area) improvement of the Deep Neural Network (DNN) accelerator that can provide up to 53%, 41%, and 63% reduction in energy, latency, and area, respectively, across a broad set of relevant AR/VR workloads. Last, we present full-system evaluation results by taking image sensing, cross-tier data transfer, and off-sensor communication into consideration.

Embedded non-volatile memories (NVMs) hold significant promise for advancing edge AI applications by offering unique advantages in near/in-memory computing-based accelerators. This invited paper delivers an in-depth review of the use cases for NVMs, emphasizing their potential to enhance area- and energy-efficiency in edge devices. We begin by outlining the latest advancements in TSMC NVM technologies and examining several NVM-based accelerator test chips enabled by the TSMC University Shuttle Program. Additionally, we delve into the tradeoffs involved in optimizing NVM devices, explore their potential for approximate computing applications, and assess the impact of NVM non-idealities on inference accuracy.

Embedded non-volatile memories (NVMs) hold significant promise for advancing edge AI applications by offering unique advantages in near/in-memory computing-based accelerators. This invited paper delivers an in-depth review of the use cases for NVMs, emphasizing their potential to enhance area- and energy-efficiency in edge devices. We begin by outlining the latest advancements in TSMC NVM technologies and examining several NVM-based accelerator test chips enabled by the TSMC University Shuttle Program. Additionally, we delve into the tradeoffs involved in optimizing NVM devices, explore their potential for approximate computing applications, and assess the impact of NVM non-idealities on inference accuracy.

STT-MRAM shows great promise for use in artificial intelligence (AI) edge devices due to its compact bitcell area and high endurance. However, it faces read challenges because of its low TMR and $\mathrm{R}_{\mathrm{p}}$. Conventional sense amplifiers have limitations in optimizing read energy and robustness while providing flexibility to exploit neural-net error tolerance. This article explores the design challenges of conventional sense amplifiers and examines how device parameters (TMR and $\mathrm{R}_{\mathrm{P}})$ impact read performances. A novel capacitive-coupling sense amplifier is introduced to offer a new design space for balancing read energy and robustness. Combining the exploitation of neural-net error tolerance with sense amplifier and device co-design, a Design-Technology-System Co-Optimization (DTSCO) approach demonstrates a read energy reduction of 27.1 % to 45.3% with minimal inference accuracy degradation in edge AI applications.

Three-dimensional heterogeneous integration (3-D-HI) has been proposed as a potential method to stack a large amount of embedded memory required in state-of-the-art compute-in-memory (CIM) artificial intelligence (AI) accelerators. While embedded nonvolatile memory, such as resistive RAM (RRAM), is a promising alternative to static random access memory (SRAM)/dynamic random access memory (DRAM) as a CIM synaptic device owing to high density, low leakage, and nondestructive read, thermal-induced device conductance drift remains a challenge. High-temperature-driven lower retention can be more significant in dense memory-logic 3-D integration due to increased volumetric power, which has not been studied in prior work. The scope of this work is to quantify the thermal impact of different 3-D-HI architectures on the reliability of 3-D-integrated binary RRAM devices for CIM applications. A device-integration-application reliability evaluation methodology is proposed, using which 3-D integration architectures and logic-memory partitioning configurations are benchmarked. Due to higher junction temperatures for memory tier in both five-tier monolithic 3-D (M3D) and five-tier through silicon via (TSV)-based 3-D compared to the 2-D baseline, the drop in inference accuracy at ten years is $\approx 80$ %. For our assumed device, integration, and application parameters, a three-tier configuration provides a balanced design option between thermal and application performance. The integrated benchmark framework is released on GitHub (https://github.com/i3dsystems/3D_CIM_thermal_v1.0) as an open-source tool for the research community.

On-device embedded artificial intelligence prefers the adaptive learning capability when deployed in the field, and thus in situ training is required. The compute-in-memory approach, which exploits the analog computation within the memory array, is a promising solution for deep neural network (DNN) on-chip acceleration. Emerging non-volatile memories are of great interest, serving as analog synapses due to their multilevel programmability. However, the asymmetry and nonlinearity in the conductance tuning remain grand challenges for achieving high in situ training accuracy. In addition, analog-to-digital converters at the edge of the memory array introduce quantization errors. In this work, we present an algorithm-hardware co-optimization to overcome these challenges. We incorporate the device/circuit non-ideal effects into the DNN propagation and weight update steps. By introducing the adaptive “momentum” in the weight update rule, in situ training accuracy on CIFAR-10 could approach its software baseline even under severe asymmetry/nonlinearity and analog-to-digital converter quantization error. The hardware performance of the on-chip training architecture and the overhead for adding “momentum” are also evaluated. By optimizing the backpropagation dataflow, 23.59 TOPS/W training energy efficiency (12× improvement compared to naïve dataflow) is achieved. The circuits that handle “momentum” introduce only 4.2% energy overhead. Our results show great potential and more relaxed requirements that enable emerging non-volatile memories for DNN acceleration on the embedded artificial intelligence platforms.

Compute-in-memory (CIM) is an attractive solution to process the extensive workloads of multiply-and-accumulate (MAC) operations in deep neural network (DNN) hardware accelerators. A simulator with options of various mainstream and emerging memory technologies, architectures, and networks can be a great convenience for fast early-stage design space exploration of CIM hardware accelerators. DNN+NeuroSim is an integrated benchmark framework supporting flexible and hierarchical CIM array design options from a device level, to a circuit level and up to an algorithm level. In this study, we validate and calibrate the prediction of NeuroSim against a 40-nm RRAM-based CIM macro post-layout simulations. First, the parameters of a memory device and CMOS transistor are extracted from the foundry’s process design kit (PDK) and employed in the NeuroSim settings; the peripheral modules and operating dataflow are also configured to be the same as the actual chip implementation. Next, the area, critical path, and energy consumption values from the SPICE simulations at the module level are compared with those from NeuroSim. Some adjustment factors are introduced to account for transistor sizing and wiring area in the layout, gate switching activity, post-layout performance drop, etc. We show that the prediction from NeuroSim is precise with chip-level error under 1% after the calibration. Finally, the system-level performance benchmark is conducted with various device technologies and compared with the results before the validation. The general conclusions stay the same after the validation, but the performance degrades slightly due to the post-layout calibration.

Compute-in-memory (CIM) is an attractive solution to process the extensive workloads of multiply-and-accumulate (MAC) operations in deep neural network (DNN) hardware accelerators. A simulator with options of various mainstream and emerging memory technologies, architectures and networks can be a great convenience for fast early-stage design space exploration of CIM accelerators. DNN+NeuroSim is an integrated benchmark framework supporting flexible and hierarchical CIM array design options from device-level, to circuit-level and up to algorithm-level. In this paper, we validate and calibrate the prediction of NeuroSim against a 40nm RRAM-based CIM macro post-layout simulations. First, the parameters of memory device and CMOS transistor are extracted from the TSMC’s PDK and employed on the NeuroSim settings; the peripheral modules and operating process are also configured to be the same as the actual chip. Next, the area, critical path and energy consumption values from the SPICE simulations at the module-level are compared with those from NeuroSim. Some adjustment factors are introduced to account for transistor sizing and wiring area in layout, gate switching activity and post-layout performance drop, etc. We show that the prediction from NeuroSim is precise with chip-level error under 1% after the calibration.

Compute-in-memory (CIM) is an attractive solution to address the “memory wall” challenges for the extensive computation in deep learning hardware accelerators. For custom ASIC design, a specific chip instance is restricted to a specific network during runtime. However, the development cycle of the hardware is normally far behind the emergence of new algorithms. Although some of the reported CIM-based architectures can adapt to different deep neural network (DNN) models, few details about the dataflow or control were disclosed to enable such an assumption. Instruction set architecture (ISA) could support high flexibility, but its complexity would be an obstacle to efficiency. In this article, a runtime reconfigurable design methodology of CIM-based accelerators is proposed to support a class of convolutional neural networks running on one prefabricated chip instance with ASIC-like efficiency. First, several design aspects are investigated: (1) the reconfigurable weight mapping method; (2) the input side of data transmission, mainly about the weight reloading; and (3) the output side of data processing, mainly about the reconfigurable accumulation. Then, a system-level performance benchmark is performed for the inference of different DNN models, such as VGG-8 on a CIFAR-10 dataset and AlexNet GoogLeNet, ResNet-18, and DenseNet-121 on an ImageNet dataset to measure the trade-offs between runtime reconfigurability, chip area, memory utilization, throughput, and energy efficiency.

submissions for the 72nd ECTC are due by