Analog AI implements the multiply-accumulate operations that dominate deep learning at the location of the weight data, offering orders of magnitude performance and energy improvements over conventional digital systems. However, translating these benefits to the Convolutional Neural Networks (CNN) widely used in applications such as image and speech processing is non-trivial. Significant reuse of both weights and activations, together with the need to extensively rearrange activations between layers, require micro-architectural solutions that span across weight mapping strategy, activation positioning, pipelining between stages, and data transport. In this paper, we describe a weight-stationary Analog AI micro-architecture for Convolutional Neural Networks (CNNs), called Row-By-Row- (RBR-) CNNs and its associated circuits and pipelines. We show a hardware demonstration of RBR-CNNs on a 14nm Analog AI inference chip with Phase Change Memory (PCM), achieving software-equivalent accuracy on the ML Perf Benchmark “Key Word Spotting” task using 4 RBR CNN layers. We show RBR CNNs using Analog AI can achieve 7x–15x latency improvements vs. high performance chips reported on ML Perf, while offering extremely high energy-efficiency – at least two orders of magnitude better than published low-power edge chip results, indicating strong applicability in embedded and mobile settings.

A Lite Bidirectional Encoder Representations from Transformers model is demonstrated on an analog inference chip fabricated at 14nm node with phase change memory. The 7.1 million unique analog weights shared across 12 layers are mapped to a single chip, accurately programmed into the conductance of 28.3 million devices, for this first analog hardware demonstration of a meaningfully large Transformer model. The implemented model achieved near iso-accuracy on the General Language Understanding Evaluation benchmark of seven tasks, despite the presence of weight-programming errors, hardware imperfections, readout noise, and error propagation. The average hardware accuracy was only 1.8% below that of the floating-point reference, with several tasks at full iso-accuracy. Careful fine-tuning of model weights using hardware-aware techniques contributes an average hardware accuracy improvement of 4.4%. Accuracy loss due to conductance drift - measured to be roughly 5% over 30 days - was reduced to less than 1% with a recalibration-based "drift compensation" technique.

Stochastic sampling is performed to reduce hardware energy consumption and prevent overfitting by reducing parameters, because not all data are required for learning. In this study, a new approach, pseudo-synaptic sampling (PS2) method, which approximates the conventional synaptic sampling machine (S2M) method through a hardware-friendly implementation while demonstrating superior efficiency, is introduced. By sampling in front of neurons rather than at each synapse, the PS2 method improves hardware energy efficiency and ensures scalability. Furthermore, it improves energy and area efficiency by eliminating the additional circuit required by other techniques, such as the random walk (RW) method previously used which requires an additional circuit to frequently charge/discharge the membrane potential. Herein, the average firing rate equation for the S2M method is modified to suit the experimental conditions of this study. Through this numerical simulations, it is confirmed that the activation function of the PS2 method aligns with that of the S2M method and verified that the PS2 method can implement stochasticity for restricted Boltzmann machine (RBM) neurons. Experimental validation of the PS2 method, compared to the RW method, for Modified National Institute of Standards and Technology database (MNIST) training and inference on field-programmable-gate-array-implemented spiking RBM chips reveals promising results. In an MNIST 100-handwritten digit experiment, the PS2 method exhibits on-chip training accuracy (92%) comparable to that of the RW method (93%). Furthermore, in the energy consumption analysis, it is shown that the PS2 method reduces power consumption by 94.94% compared to the RW method, highlighting its enhanced power efficiency due to a reduced number of circuit elements. In the investigation into the impact of increasing the frequency at which random bits are generated, it is shown that the RW method experiences accuracy degradation even with slight increases, whereas the PS2 method maintains accuracy over significantly longer periods. This enables further power reduction by allowing for a longer period during random bit generation. In this study, a foundation is laid for maximizing the energy efficiency of spiking neural network processors by optimizing internal noise generation mechanisms.

Emerging non-volatile memory (NVM) devices, such as STT-MRAM, PCM, RRAM, have been explored for embedded memory and storage applications to replace CMOS-based SRAM/DRAM and Flash devices. Recently, many of these memory devices have been utilized for new computing paradigms beyond Boolean logic and von Neumann architectures. For example, in-memory analog computing reduces data movement between computing and memory units and exploits the intrinsic parallelism in memory arrays. It finds a natural application in deep neural network (DNN) accelerators by implementing high-throughput high-efficiency multiply accumulate (MAC) operations. Here the conductance of memory devices in a crossbar array represents DNN weights and the activations are encoded in input electrical signals (e.g., pulse height or duration). The MAC operation is conducted via the Ohm’s law (multiplication between voltage and conductance) and Kirchhoff’s law (accumulate via current summation) at constant time even for very large networks. DNN has surpassed human performance in various AI applications, e.g., image classification, natural language processing, etc. While general-purpose CPU/GPU and special-purpose digital accelerators provide current and near-term DNN hardware, there are longer-term opportunities for analog DNN accelerators based on emerging memory devices to achieve significantly higher performance and energy-efficiency. At the same time, analog accelerators impose new requirements on these devices beyond traditional memory applications, e.g., analog tunability, gradual and symmetric weight modulation, high precision, etc. Memory devices with analog nature in their physical mechanisms (e.g., filament growth in RRAM) may be optimized to meet these requirements, while some abrupt and asymmetric characteristics (e.g., filament rupture) present challenges. Increasingly large neural network models have been demonstrated on these memory arrays designed as analog accelerators, but they are still orders of magnitude smaller than state-of-the-art DNN models. While analog accelerators enable massively parallel computation, they are also susceptible to unique challenges in analog devices and circuitry (e.g., device variability, circuit noise), which may degrade network performance (e.g., accuracy). To benefit from the massively parallel MAC operation in analog memory arrays, these arrays need to be large enough to efficiently map the layers in modern DNN models. Among emerging NVM devices, PCM has the advantages of maturity and the availability of large-scale arrays, but also face some challenges in device characteristics, e.g., conductance drift, asymmetry, and noise. PCM-based analog DNN accelerators have been demonstrated at advanced technology node with millions of devices and achieved iso-accuracy on increasingly large network models. These accelerators integrate highly efficient analog PCM tiles for MAC operations with advanced CMOS circuitry for auxiliary digital functions. While material/device engineering continues to be explored to improve the analog properties of PCM devices, design and operation innovations can also help to improve the performance of PCM-based DNN weights, e.g., multiple-device-per-weight design, close-loop tunning. In addition, circuit innovations are essential for analog accelerator performance. Fig. 1 shows a 14nm PCM-based DNN inference accelerator, which incorporate design techniques such as 4-PCM weight units, 2D mesh for tile-to-tile communication, pulse-duration-based coding, etc. On top of technology and design innovations, some DNN models can also be modified to be more resilient against hardware imperfection and noise. PCM-based analog accelerators have achieved iso-accuracy on large DNN models with millions of weights. This talk will discuss the progress that we have achieved on PCM-based analog DNN inference accelerators, the challenges of PCM materials and devices, and promising solutions in technology and design. Figure 1

Efficiently solving combinatorial optimization problems (COPs) such as Max-Cut is challenging because the resources required increase exponentially with the problem size. This study proposes a hardware-friendly method for solving the Max-Cut problem by implementing a spiking neural network (SNN)-based Boltzmann machine (BM) in neuromorphic hardware systems. To implement the hardware-oriented version of the spiking Boltzmann machine (sBM), the stochastic dynamics of leaky integrate-and-fire (LIF) neurons with random walk noise are analyzed, and an innovative algorithm based on overlapping time windows is proposed. The simulation results demonstrate the effective convergence and high accuracy of the proposed method for large-scale Max-Cut problems. The proposed method is validated through successful hardware implementation on a 6-transistor/2-resistor (6T2R) neuromorphic chip with phase change memory (PCM) synapses. In addition, as an expansion of the algorithm, several annealing techniques and bias split methods are proposed to improve convergence, along with circuit design ideas for efficient evaluation of sampling convergence using cell arrays and spiking systems. Overall, the results of the proposed methods demonstrate the potential of energy-efficient and hardware-implementable approaches using SNNs to solve COPs. To the best of the author's knowledge, this is the first study to solve the Max-Cut problem using an SNN neuromorphic hardware chip.

Models of artificial intelligence (AI) that have billions of parameters can achieve high accuracy across a range of tasks 1 , 2 , but they exacerbate the poor energy efficiency of conventional general-purpose processors, such as graphics processing units or central processing units. Analog in-memory computing (analog-AI) 3 – 7 can provide better energy efficiency by performing matrix–vector multiplications in parallel on ‘memory tiles’. However, analog-AI has yet to demonstrate software-equivalent (SW eq ) accuracy on models that require many such tiles and efficient communication of neural-network activations between the tiles. Here we present an analog-AI chip that combines 35 million phase-change memory devices across 34 tiles, massively parallel inter-tile communication and analog, low-power peripheral circuitry that can achieve up to 12.4 tera-operations per second per watt (TOPS/W) chip-sustained performance. We demonstrate fully end-to-end SW eq accuracy for a small keyword-spotting network and near-SW eq accuracy on the much larger MLPerf 8 recurrent neural-network transducer (RNNT), with more than 45 million weights mapped onto more than 140 million phase-change memory devices across five chips.

Analog Non-Volatile Memory-based accelerators offer high-throughput and energy-efficient Multiply-Accumulate operations for the large Fully-Connected layers that dominate Transformer-based Large Language Models. We describe architectural, wafer-scale testing, chip-demo, and hardware-aware training efforts towards such accelerators, and quantify the unique raw-throughput and latency benefits of Fully-(rather than Partially-) Weight-Stationary systems.

Analog non-volatile memory (NVM)-based accelerators for Deep Neural Networks (DNNs) can achieve high-throughput and energy-efficient multiply-accumulate (MAC) operations by taking advantage of massively parallelized analog compute, implemented with Ohm's law and Kirchhoff's current law on arrays of resistive memory devices. Competitive end-to-end DNN accuracies can be obtained, provided that weights are accurately programmed onto NVM devices and MAC operations are sufficiently linear. In this paper, we report architectural and circuit advances for such Analog NVM-based accelerators. We describe a highly heterogeneous and programmable accelerator architecture for DNN inference that combines analog NVM memory-array “Tiles” for weight-stationary, energy-efficient MAC operations, together with heterogeneous special-function Compute-Cores for auxiliary digital computation. Massively parallel vectors of neuron-activation data are exchanged over short distances using a dense and efficient circuit-switched 2D mesh, enabling a wide range of DNN workloads, including CNNs, LSTMs, and Transformers. We also show a 14-nm inference chip consisting of multiple $\mathbf{512}\times \mathbf{512}$ arrays of Phase Change Memory (PCM) devices which implements multiple DNN benchmarks using such a circuit-switched 2D mesh.

Analog non-volatile memory (NVM)-based accelerators for deep neural networks implement multiply-accumulate (MAC) operations – in parallel, on large arrays of resistive devices – by using Ohm’s law and Kirchhoff’s current law. By completely avoiding weight motion, such fully weight-stationary systems can offer a unique combination of low latency, high throughput, and high energy-efficiency (e.g., high TeraOPS/W). Yet since most Deep Neural Networks (DNNs) require only modest (e.g., 4-bit) precision in synaptic operations, such systems can still deliver “software-equivalent” accuracies on a wide range of models. We describe a 14-nm inference chip, comprising multiple 512×512 arrays of Phase Change Memory (PCM) devices, which can deliver software-equivalent inference accuracy for MNIST handwritten-digit recognition and recurrent LSTM benchmarks, and discuss various PCM challenges such as conductance drift and noise.

A fully silicon‐integrated restricted Boltzmann machine (RBM) with an event‐driven contrastive divergence (eCD) training algorithm is implemented using novel stochastic leaky integrate‐and‐fire (LIF) neuron circuits and six‐transistor/2‐PCM‐resistor (6T2R) synaptic unit cells on 90 nm CMOS technology. To elaborate, designed a bidirectional, asynchronous, and parallel pulse‐signaling scheme over an analog‐weighted phase‐change memory (PCM) synapse array to enable spike‐timing‐dependent plasticity (STDP) as a local weight update rule based on eCD is designed. Building upon the initial version of this work, significantly more experimental details are added, such as the on‐chip characterization results of LIF and backward‐LIF (BLIF) and stochasticity of our random walk circuitry. The experimental characterization of these on‐chip stochastic neuron circuits shows a reasonable symmetricity between LIF and BLIF as well as the necessary stochasticity for spiking RBM operation. Fully hardware‐based image classification recorded 93% on‐chip training accuracy from 100 handwritten MNIST digit images. In addition, we experimentally demonstrated the generative characteristics of the RBM by reconstructing partial patterns on hardware. As each synapse and neuron execute its computations in an asynchronous and fully parallel fashion, the chip can perform data‐intensive machine learning (ML) tasks in a power‐efficient manner and take advantage of the sparseness of spiking.

Analog non-volatile memory (NVM)-based accelerators for deep neural networks perform high-throughput and energy-efficient multiply-accumulate (MAC) operations (e.g., high TeraOPS/W) by taking advantage of massively parallelized analog MAC operations, implemented with Ohm’s law and Kirchhoff’s current law on array-matrices of resistive devices. While the wide-integer and floating-point operations offered by conventional digital CMOS computing are much more suitable than analog computing for conventional applications that require high accuracy and true reproducibility, deep neural networks can still provide competitive end-to-end results even with modest (e.g., 4-bit) precision in synaptic operations. In this paper, we describe a 14-nm inference chip, comprising multiple 512$\times$ 512 arrays of Phase Change Memory (PCM) devices, which can deliver software-equivalent inference accuracy for MNIST handwritten-digit recognition and recurrent LSTM benchmarks, by using compensation techniques to finesse analog-memory challenges such as conductance drift and noise. We also project accuracy for Natural Language Processing (NLP) tasks performed with a state-of-art large Transformer-based model, BERT, when mapped onto an extended version of this same fundamental chip architecture.

Hardware acceleration of deep learning using analog non-volatile memory (NVM) requires large arrays with high device yield, high accuracy Multiply-ACcumulate (MAC) operations, and routing frameworks for implementing arbitrary deep neural network (DNN) topologies. In this article, we present a 14-nm test-chip for Analog AI inference—it contains multiple arrays of phase change memory (PCM)-devices, each array capable of storing 512 $\times $ 512 unique DNN weights and executing massively parallel MAC operations at the location of the data. DNN excitations are transported across the chip using a duration representation on a parallel and reconfigurable 2-D mesh. To accurately transfer inference models to the chip, we describe a closed-loop tuning (CLT) algorithm that programs the four PCM conductances in each weight, achieving <3% average weight-error. A row-wise programming scheme and associated circuitry allow us to execute CLT on up to 512 weights concurrently. We show that the test chip can achieve near-software-equivalent accuracy on two different DNNs. We demonstrate tile-to-tile transport with a fully-on-chip two-layer network for MNIST (accuracy degradation ~0.6%) and show resilience to error propagation across long sequences (up to 10 000 characters) with a recurrent long short-term memory (LSTM) network, implementing off-chip activation and vector-vector operations to generate recurrent inputs used in the next on- chip MAC.

A neuromorphic electric system includes a network of plural neuron circuits connected in series and in parallel to form plural layers. Each of the plural neuron circuits includes: a soma circuit that stores a charge supplied thereto and outputs a spike signal; and plural synapse circuits that supply a charge to the soma circuit according to a spike signal fed to the synapse circuits, a number of the plural synapse circuits being one more than a number of plural neuron circuits in a prior layer outputting the spike signal to the synapse circuits. One of the plural synapse circuits supplies a charge to the soma circuit in response to receiving a series of pulse signals, and the others of the plural synapse circuits supply a charge to the soma circuit in response to receiving a spike signal from corresponding neuron circuits in the prior layer.

Phase change memory (PCM) is arguably one of the most promising non-volatile memories which can be used in neuromorphic applications. When we use PCM as analog synaptic elements, non-ideality impact should be carefully taken into account. In this paper, we investigate the impact of such non-ideality items as nonlinear weight update and resistance drift on training accuracy in a spiking restricted Boltzmann machine (RBM). In addition to ideal PCM model, actual measurement-based PCM characteristics is used for this study. The resistance drift affects training accuracy, especially in a widely distributed read interval time caused by spatial and temporal sparse spike activities during training in spiking RBM. Our simulation results show that the training accuracy worsens with the increase of the weight-update nonlinearity or the resistance-drift coefficient. However, the results also suggest we can expect more than 8.95% improvement in training accuracy. This potential improvement will be possible if we use linearity-improved confined PCM cells whose median resistance drift coefficient is 0.005, compared to the resistance drift coefficient of more than 0.02 in typical existing PCM cells.

A fully silicon-integrated restricted Boltzmann machine (RBM) with event-driven contrastive divergence (eCD) algorithm is implemented using novel stochastic leaky integrate-and-fire (LIF) neuron circuits and 6-transistor/2- PCM-resistor (6T2R) unit cells on 90-nm CMOS technology. A bidirectional asynchronous spiking signaling scheme over an analog-weighted phase change memory (PCM) crossbar enables spike-timing-dependent plasticity (STDP) as a local weight update rule. This results in concurrent massively- parallel neuronal computation for low-power on-chip training and inference. Experimental image classification using 100 handwritten digit images from the MNIST database demonstrates 92% training accuracy. SPICE simulation abstracted from the fabricated design indicates 8.95 power. A projection to 28-nm technology gives 5.39 pJ per synaptic operation.

To accelerate performance per power in demanding machine learning applications in data centers, recently graphics processing units (GPU), field-programmable gate array (FPGA) and applicationspecific integrated circuit (ASIC) are broadly utilized by equipping massively parallel digital multiply-accumulators, where multiply-accumulation is an arithmetic bottleneck in software workloads using neural networks. Some of intelligent edge devices also become equipping ASIC or IP cores including such massively parallel multiply-accumulators optimized for specific machine learning applications at low-power consumption. This presentation introduces our current research activities in analog-based machine learning accelerators and analog-based spiking neural network processors aiming at high performance per power in machine learning applications by taking advantage of high-speed and low-power analog multiply-accumulation arithmetic over densely-packed synaptic non-volatile memory (NVM) resistive device arrays.