A vertically integrated, end-to-end, research prototype system combines 288 NorthPole neural inference accelerator cards, offline training algorithms, a high-performance runtime stack, and a containerized inference pipeline to deliver a scalable and efficient cloud inference service. The system delivers 115 peta-ops at 4-bit integer precision and 3.7 PB/s of memory bandwidth across 18 2U servers, while consuming only 30 kW of power and weighing 730 kg in a 0.67 m^2 42U rack footprint. The system can run 3 simultaneous instances of the 8-billion-parameter open-source IBM Granite-3.3-8b-instruct model at 2,048 context length with 28 simultaneous users and a per-user inter-token latency of 2.8 ms. The system is scalable, modular, and reconfigurable, supporting various model sizes and context lengths, and is ideal for deploying agentic workflows for enterprise AI applications in existing data center (cloud, on-prem) environments. For example, the system can support 18 instances of a 3-billion-parameter model or a single instance of a 70-billion-parameter model.

For a 3-billion-parameter LLM, a research prototype inference appliance with 16 IBM AIU NorthPole processors delivers a massive 28,356 tokens / second of system throughput and sub-1 ms / token (per-user) latency while consuming merely 672 W for 16 NorthPole cards in a compact 2U form factor. With a focus on low latency and high energy efficiency, when NorthPole (in 12 nm) is compared to a suite of GPUs (in 7 / 5 / 4 nm) at various power consumptions, at the lowest GPU latency, NorthPole provides 72.7. better energy metric (tokens / second / W) while providing better latency.

Stochastic sampling is performed to reduce hardware energy consumption and prevent overfitting by reducing parameters, because not all data are required for learning. In this study, a new approach, pseudo-synaptic sampling (PS2) method, which approximates the conventional synaptic sampling machine (S2M) method through a hardware-friendly implementation while demonstrating superior efficiency, is introduced. By sampling in front of neurons rather than at each synapse, the PS2 method improves hardware energy efficiency and ensures scalability. Furthermore, it improves energy and area efficiency by eliminating the additional circuit required by other techniques, such as the random walk (RW) method previously used which requires an additional circuit to frequently charge/discharge the membrane potential. Herein, the average firing rate equation for the S2M method is modified to suit the experimental conditions of this study. Through this numerical simulations, it is confirmed that the activation function of the PS2 method aligns with that of the S2M method and verified that the PS2 method can implement stochasticity for restricted Boltzmann machine (RBM) neurons. Experimental validation of the PS2 method, compared to the RW method, for Modified National Institute of Standards and Technology database (MNIST) training and inference on field-programmable-gate-array-implemented spiking RBM chips reveals promising results. In an MNIST 100-handwritten digit experiment, the PS2 method exhibits on-chip training accuracy (92%) comparable to that of the RW method (93%). Furthermore, in the energy consumption analysis, it is shown that the PS2 method reduces power consumption by 94.94% compared to the RW method, highlighting its enhanced power efficiency due to a reduced number of circuit elements. In the investigation into the impact of increasing the frequency at which random bits are generated, it is shown that the RW method experiences accuracy degradation even with slight increases, whereas the PS2 method maintains accuracy over significantly longer periods. This enables further power reduction by allowing for a longer period during random bit generation. In this study, a foundation is laid for maximizing the energy efficiency of spiking neural network processors by optimizing internal noise generation mechanisms.

We present preliminary results demonstrating AI (artificial intelligence) inference using the IBM AIU NorthPole Chip [1], [2] incorporated into a compact, rugged 3U VPX form factor module (NP-VPX) [3]. NP-VPX allows NorthPole to be used in edge applications with stringent cooling requirements, high-speed switch fabrics, and rugged environments. NP-VPX processes 965 frames per second (fps) with a Yolo-v4 network with 640x640 pixel images at 73.5 W at full-precision accuracy, achieving 13.2 frames / J (fps / W). NP-VPX processes over 40,300 fps with a ResNet-50 network with 224x224 pixel images at 65.9 W at full-precision accuracy, achieving 611 frames / J.

Efficiently solving combinatorial optimization problems (COPs) such as Max-Cut is challenging because the resources required increase exponentially with the problem size. This study proposes a hardware-friendly method for solving the Max-Cut problem by implementing a spiking neural network (SNN)-based Boltzmann machine (BM) in neuromorphic hardware systems. To implement the hardware-oriented version of the spiking Boltzmann machine (sBM), the stochastic dynamics of leaky integrate-and-fire (LIF) neurons with random walk noise are analyzed, and an innovative algorithm based on overlapping time windows is proposed. The simulation results demonstrate the effective convergence and high accuracy of the proposed method for large-scale Max-Cut problems. The proposed method is validated through successful hardware implementation on a 6-transistor/2-resistor (6T2R) neuromorphic chip with phase change memory (PCM) synapses. In addition, as an expansion of the algorithm, several annealing techniques and bias split methods are proposed to improve convergence, along with circuit design ideas for efficient evaluation of sampling convergence using cell arrays and spiking systems. Overall, the results of the proposed methods demonstrate the potential of energy-efficient and hardware-implementable approaches using SNNs to solve COPs. To the best of the author's knowledge, this is the first study to solve the Max-Cut problem using an SNN neuromorphic hardware chip.

The Deep Neural Network (DNN) era was ushered in by the triad of algorithms, big data, and more powerful hardware processors for training large-scale neural networks. Now, the ubiquitous deployment of DNNs for neural inference in edge, embedded, and data center applications demands more power-efficient hardware processors, while attaining increasingly higher computational performance. To address this Inference Challenge, we developed the NorthPole Architecture and implemented a NorthPole Chip instantiation [1, 2].

Computing, since its inception, has been processor-centric, with memory separated from compute. Inspired by the organic brain and optimized for inorganic silicon, NorthPole is a neural inference architecture that blurs this boundary by eliminating off-chip memory, intertwining compute with memory on-chip, and appearing externally as an active memory chip. NorthPole is a low-precision, massively parallel, densely interconnected, energy-efficient, and spatial computing architecture with a co-optimized, high-utilization programming model. On the ResNet50 benchmark image classification network, relative to a graphics processing unit (GPU) that uses a comparable 12-nanometer technology process, NorthPole achieves a 25 times higher energy metric of frames per second (FPS) per watt, a 5 times higher space metric of FPS per transistor, and a 22 times lower time metric of latency. Similar results are reported for the Yolo-v4 detection network. NorthPole outperforms all prevalent architectures, even those that use more-advanced technology processes.

A fully silicon‐integrated restricted Boltzmann machine (RBM) with an event‐driven contrastive divergence (eCD) training algorithm is implemented using novel stochastic leaky integrate‐and‐fire (LIF) neuron circuits and six‐transistor/2‐PCM‐resistor (6T2R) synaptic unit cells on 90 nm CMOS technology. To elaborate, designed a bidirectional, asynchronous, and parallel pulse‐signaling scheme over an analog‐weighted phase‐change memory (PCM) synapse array to enable spike‐timing‐dependent plasticity (STDP) as a local weight update rule based on eCD is designed. Building upon the initial version of this work, significantly more experimental details are added, such as the on‐chip characterization results of LIF and backward‐LIF (BLIF) and stochasticity of our random walk circuitry. The experimental characterization of these on‐chip stochastic neuron circuits shows a reasonable symmetricity between LIF and BLIF as well as the necessary stochasticity for spiking RBM operation. Fully hardware‐based image classification recorded 93% on‐chip training accuracy from 100 handwritten MNIST digit images. In addition, we experimentally demonstrated the generative characteristics of the RBM by reconstructing partial patterns on hardware. As each synapse and neuron execute its computations in an asynchronous and fully parallel fashion, the chip can perform data‐intensive machine learning (ML) tasks in a power‐efficient manner and take advantage of the sparseness of spiking.

Phase change memory (PCM) is arguably one of the most promising non-volatile memories which can be used in neuromorphic applications. When we use PCM as analog synaptic elements, non-ideality impact should be carefully taken into account. In this paper, we investigate the impact of such non-ideality items as nonlinear weight update and resistance drift on training accuracy in a spiking restricted Boltzmann machine (RBM). In addition to ideal PCM model, actual measurement-based PCM characteristics is used for this study. The resistance drift affects training accuracy, especially in a widely distributed read interval time caused by spatial and temporal sparse spike activities during training in spiking RBM. Our simulation results show that the training accuracy worsens with the increase of the weight-update nonlinearity or the resistance-drift coefficient. However, the results also suggest we can expect more than 8.95% improvement in training accuracy. This potential improvement will be possible if we use linearity-improved confined PCM cells whose median resistance drift coefficient is 0.005, compared to the resistance drift coefficient of more than 0.02 in typical existing PCM cells.

A fully silicon-integrated restricted Boltzmann machine (RBM) with event-driven contrastive divergence (eCD) algorithm is implemented using novel stochastic leaky integrate-and-fire (LIF) neuron circuits and 6-transistor/2- PCM-resistor (6T2R) unit cells on 90-nm CMOS technology. A bidirectional asynchronous spiking signaling scheme over an analog-weighted phase change memory (PCM) crossbar enables spike-timing-dependent plasticity (STDP) as a local weight update rule. This results in concurrent massively- parallel neuronal computation for low-power on-chip training and inference. Experimental image classification using 100 handwritten digit images from the MNIST database demonstrates 92% training accuracy. SPICE simulation abstracted from the fabricated design indicates 8.95 power. A projection to 28-nm technology gives 5.39 pJ per synaptic operation.

Neuromorphic circuits with nonvolatile memory crossbar arrays can train and inference neural networks in a highly power-efficient manner, which can be a solution to overcome the von Neumann bottle-neck. This paper proposes a scalable multi-core spiking neuromorphic system architecture that can support a large-scale multi-layer neural network larger than a network supported by a computing system with a single neuromorphic circuit core. To simplify the inter-core communication, neuromorphic cores communicate only by sending and receiving spikes. Deep networks can be easily formed on this architecture by connecting multiple cores. The neuromorphic cores are trained on-chip by backpropagation, which is a well-known and sophisticated algorithm for training neural networks in software. We made modifications to the traditional backpropagation algorithm to propagate errors and update weights by spikes on the spiking neuromorphic cores of a computing system using our architecture. The proposed algorithm was evaluated by an spike event-based neuromorphic circuit simulator using three datasets. Cancer1 and Thyroid1 were used for a small network evaluation, which results showed better test error than previous studies, and MNIST was used to evaluate a large realistic neural network.

Neuromorphic computing using analog non-volatile memory (NVM) devices has been the subject of various studies due to its potential ability to achieve extremely low power consumption less than that of traditional von Neumann architecture. However, using NVM devices, such as phase change memory (PCM) and resistive-RAM devices, presents various challenges, such as limitations in the number of conductance steps and device variability resulting from process variation and electro/thermo-dynamics. Limitations in the number of conductance steps and device variability could reduce the accuracy of neural network training. It is necessary to quantitatively analyze the effect of the number of conductance steps and synaptic device variability on the accuracy of neural network training and assess requirements for NVM devices to make NVM-based neuromorphic computing successful. We conducted the analysis using simulations focusing on a spiking neural network (SNN) based restricted Boltzmann machine (RBM) with PCM devices using the PCM-refresh method. The results of our quantitative simulation, which used the MNIST dataset, showed that having more than 500 conductance steps achieves comparable performance to that when there are more than 1000 conductance steps. We also found that less than 10% conductance update variation in the synaptic devices is required to achieve the comparable accuracy with the no variation case. These results can provide guidelines for designing and optimizing a synaptic device for realizing NVM-based neuromorphic computing.

In extremely energy-efficient neuromorphic computing using analog non-volatile memory (NVM) devices, device variability arises due to process variation and electro/thermo-dynamics of NVM devices, such as phase change memory and resistive-RAM. Thus, for realizing NVM-based neuromorphic computing, it is important to quantitatively analyze the impact of synaptic device variability on neural network training accuracy and assess requirements for NVM devices. We investigated the analysis using simulations focusing on a spiking neural network (SNN)-based restricted Boltzmann machine (RBM). MNIST dataset simulation results revealed that more than 500 steps of conductance achieve comparable performance to the previous study of software-based simulation on SNN-based RBM. We also observed that at least a less than 10% of variation in conductance update for each synaptic device is required for achieving comparable performance to the result with no variation. These results provide baselines for designing and optimizing the characteristics of NVM devices.

Phase change memory (PCM) is being explored as a synaptic nanodevice for scalable and low-power neuromorphic circuits. We present a novel and lightweight method to refresh PCM cells after they saturate at their maximum conductance during the learning process. Our learning system is an event-based Restricted Boltzmann Machine with Spike Time Dependent Plasticity update rule using a modified contrastive divergence algorithm. By using our event-based neuromorphic circuit simulator and the MNIST handwritten digit dataset, we show that our refresh method reduces power consumption by decreasing the number of SET and RESET programming pulses while maintaining high learning accuracy.

A systolic array is known as a parallel hardware architecture applicable to a wide range of applications. Naive implementations, however, can lead to inefficient resource usage and low power performance. In this paper, we discuss two techniques for improving the hardware resource usage: flexible multi-threading and dummy data padding. The design was implemented to accelerate a pair-HMM algorithm on an FPGA with the IBM POWER8 CAPI (Coherent Accelerator Processor Interface) feature. The CAPI feature simplifies the software design for driving the FPGA accelerator. Our experimental result indicates that the implemented FPGA accelerator executing the pair-HMM algorithm achieves 33x higher power performance than a POWER8 processor chip executing the same algorithm.

As the increase of clock frequencies has slowed, special purpose hardware circuits are becoming increasingly important to accelerate the performance of computing systems. In this context, FPGAs offer advantages over hardcoded ASICs, since FPGAs allow us to use the entire chip to implement optimized algorithms for specific inputs by reconfiguring the circuits at runtime. For example, relational database systems are typically implemented with multiple algorithms for each relational algebraic operation and the one that is expected to process a given query most quickly can be selected as needed. However, previous research on FPGA acceleration for databases has not paid much attention to such algorithm selection. This paper describes an FPGA equi-joiner that switches between two equi-join algorithms, a hash join and a sort-merge join, to fully allocate the FPGA’s resources to one algorithm at a time. Our implementation of each algorithm takes advantage of the fact that it can use most of the hardware resources on an FPGA to maximize the size of a key component, a hash table for the hash join and a sort-merge tree for the sort-merge join, which is critical for the join performance. Our experimental results have shown that a simple mathematical model can be used to estimate the execution time of each algorithm for a given data size to select the algorithm appropriately. Keywords-FPGA; Reconfiguration; Relational Database; Equi Join; Hash Join; Sort Merge Join