This is a case study of how NorthPole, a neural inference accelerator with a highly novel architecture, was designed, verified, and fabricated successfully in first-silicon using a horizontally integrated design workflow centered on a cycle-accurate functional simulator called the NorthPole Validator. The Validator is NorthPole’s digital twin. The study demonstrates how the Validator’s scalable network-of-queues structure, consisting of approximately 26,000 nodes and 415,000 queues, let it easily pivot from architecture validation and compiler codesign, to logic and physical design verification, post-silicon testing, and software ecosystem development.

A vertically integrated, end-to-end, research prototype system combines 288 NorthPole neural inference accelerator cards, offline training algorithms, a high-performance runtime stack, and a containerized inference pipeline to deliver a scalable and efficient cloud inference service. The system delivers 115 peta-ops at 4-bit integer precision and 3.7 PB/s of memory bandwidth across 18 2U servers, while consuming only 30 kW of power and weighing 730 kg in a 0.67 m^2 42U rack footprint. The system can run 3 simultaneous instances of the 8-billion-parameter open-source IBM Granite-3.3-8b-instruct model at 2,048 context length with 28 simultaneous users and a per-user inter-token latency of 2.8 ms. The system is scalable, modular, and reconfigurable, supporting various model sizes and context lengths, and is ideal for deploying agentic workflows for enterprise AI applications in existing data center (cloud, on-prem) environments. For example, the system can support 18 instances of a 3-billion-parameter model or a single instance of a 70-billion-parameter model.

We show for invertible problems that transform data from a source domain (for example, Logic Condition Tables (LCTs)) to a destination domain (for example, Hardware Description Language (HDL) code), an approach of using Large Language Models (LLMs) as a lossless encoder from source to destination followed by as a lossless decoder back to the source, comparable to lossless compression in information theory, can mitigate most of the LLM drawbacks of hallucinations and omissions. Specifically, using LCTs as inputs, we generate the full HDL for a two-dimensional network-on-chip router (13 units, 1500-2000 lines of code) using seven different LLMs, reconstruct the LCTs from the auto-generated HDL, and compare the original and reconstructed LCTs. This approach yields significant productivity improvements, not only confirming correctly generated LLM logic and detecting incorrectly generated LLM logic but also assisting developers in finding design specification errors.

For a 3-billion-parameter LLM, a research prototype inference appliance with 16 IBM AIU NorthPole processors delivers a massive 28,356 tokens / second of system throughput and sub-1 ms / token (per-user) latency while consuming merely 672 W for 16 NorthPole cards in a compact 2U form factor. With a focus on low latency and high energy efficiency, when NorthPole (in 12 nm) is compared to a suite of GPUs (in 7 / 5 / 4 nm) at various power consumptions, at the lowest GPU latency, NorthPole provides 72.7. better energy metric (tokens / second / W) while providing better latency.

We present preliminary results demonstrating AI (artificial intelligence) inference using the IBM AIU NorthPole Chip [1], [2] incorporated into a compact, rugged 3U VPX form factor module (NP-VPX) [3]. NP-VPX allows NorthPole to be used in edge applications with stringent cooling requirements, high-speed switch fabrics, and rugged environments. NP-VPX processes 965 frames per second (fps) with a Yolo-v4 network with 640x640 pixel images at 73.5 W at full-precision accuracy, achieving 13.2 frames / J (fps / W). NP-VPX processes over 40,300 fps with a ResNet-50 network with 224x224 pixel images at 65.9 W at full-precision accuracy, achieving 611 frames / J.

The Deep Neural Network (DNN) era was ushered in by the triad of algorithms, big data, and more powerful hardware processors for training large-scale neural networks. Now, the ubiquitous deployment of DNNs for neural inference in edge, embedded, and data center applications demands more power-efficient hardware processors, while attaining increasingly higher computational performance. To address this Inference Challenge, we developed the NorthPole Architecture and implemented a NorthPole Chip instantiation [1, 2].

Computing, since its inception, has been processor-centric, with memory separated from compute. Inspired by the organic brain and optimized for inorganic silicon, NorthPole is a neural inference architecture that blurs this boundary by eliminating off-chip memory, intertwining compute with memory on-chip, and appearing externally as an active memory chip. NorthPole is a low-precision, massively parallel, densely interconnected, energy-efficient, and spatial computing architecture with a co-optimized, high-utilization programming model. On the ResNet50 benchmark image classification network, relative to a graphics processing unit (GPU) that uses a comparable 12-nanometer technology process, NorthPole achieves a 25 times higher energy metric of frames per second (FPS) per watt, a 5 times higher space metric of FPS per transistor, and a 22 times lower time metric of latency. Similar results are reported for the Yolo-v4 detection network. NorthPole outperforms all prevalent architectures, even those that use more-advanced technology processes.

IBM's brain-inspired processor is a massively parallel neural network inference engine containing 1 million spiking neurons and 256 million low-precision synapses. Now, after a decade of fundamental research spanning neuroscience, architecture, chips, systems, software, and algorithms, IBM has delivered the largest neurosynaptic computer ever built.

Word2vec, like other ways of creating word embeddings from a text corpus, has shown that interesting mathematical properties exist between the resulting word vectors. Word similarities as well as relationships can be discovered by determining which words are nearby in vector space and performing simple vector operations. In this work, IBM's TrueNorth Neurosynaptic System was used to implement massively-parallel word similarity computations using a large network of hardware spiking neurons. A 4-bit vector-matrix multiplication engine was implemented on TrueNorth in order to accommodate a word vector dictionary of 95,000 words trained on Wikipedia text, and it successfully performs word similarity searches using that dictionary while utilizing 3,991 cores out of the 4,096 available on TrueNorth and consuming less than 70 mW of power.

This paper describes a fully spike-based neural network for optical flow estimation from dynamic vision sensor data. A low power embedded implementation of the method, which combines the asynchronous time-based image sensor with IBM's TrueNorth Neurosynaptic System, is presented. The sensor generates spikes with submillisecond resolution in response to scene illumination changes. These spike are processed by a spiking neural network running on TrueNorth with a 1-ms resolution to accurately determine the order and time difference of spikes from neighbouring pixels, and therefore infer the velocity. The spiking neural network is a variant of the Barlow Levick method for optical flow estimation. The system is evaluated on two recordings for which ground truth motion is available, and achieves an average endpoint error of 11% at an estimated power budget of under 80 mW for the sensor and computation.

Deep neural networks (DNN) have been shown to be very effective at solving challenging problems in several areas of computing, including vision, speech, and natural language processing. However, traditional platforms for implementing these DNNs are often very power hungry, which has lead to significant efforts in the development of configurable platforms capable of implementing these DNNs efficiently. One of these platforms, the IBM TrueNorth processor, has demonstrated very low operating power in performing visual computing and neural network classification tasks in real-time. The neuron computation, synaptic memory, and communication fabrics are all configurable, so that a wide range of network types and topologies can be mapped to TrueNorth. This reconfigurability translates into the capability to support a wide range of low-power functions in addition to feed-forward DNN classifiers, including for example, the audio processing functions presented here.In this work, we propose an end-to-end audio processing pipeline that is implemented entirely on a TrueNorth processor and designed to specifically leverage the highly-parallel, low-precision computing primitives TrueNorth offers. As part of this pipeline, we develop an audio feature extractor (LATTE) designed for implementation on TrueNorth, and explore the tradeoffs among several design variants in terms of accuracy, power, and performance. We customize the energy-efficient deep neuromorphic networks structures that our design utilizes as the classifier and show how classifier parameters can trade between power and accuracy. In addition to enabling a wide range of diverse functions, the reconfigurability of TrueNorth enables re-training and re-programming the system to satisfy varying energy, speed, area, and accuracy requirements. The resulting system's end-to-end power consumption can be as low as $14.43\text{mW}$ , which would give up to 100 hours of continuous usage with button cell batteries (CR3023 $1.5\; \text{Whr}$ ) or 450 hours with cellphone batteries (iPhone 6s $6.55\; \text{Whr}$ ).

With recent advances in silicon technology, previously intractable Deep Neural Network (DNN) solutions to complex visual, auditory, and other sensory perception problems are now practical for real-time, energy constrained systems. One such advancement is IBM's TrueNorth neurosynaptic processor, containing 1 million neurons and 256 million synapses, consuming 65mW of power, and capable of operating in real-time for a variety of applications. In this work, we explore how auditory features can be extracted on the TrueNorth processor using low numerical precision while maintaining algorithmic fidelity for DNN based spoken digit recognition on isolated words from the TIDIGITS dataset. Further, we show that our Low-power Audio Transform with TrueNorth Ecosystem (LATTE) is capable of achieving a 24× reduction in energy for feature extraction over a baseline FPGA implementation using standard MFCC audio features, while only incurring a 3 - 6% accuracy penalty.

In recent years the field of neuromorphic low-power systems gained significant momentum, spurring brain-inspired hardware systems which operate on principles that are fundamentally different from standard digital computers and thereby consume orders of magnitude less power. However, their wider use is still hindered by the lack of algorithms that can harness the strengths of such architectures. While neuromorphic adaptations of representation learning algorithms are now emerging, the efficient processing of temporal sequences or variable length-inputs remains difficult, partly due to challenges in representing and configuring the dynamics of spiking neural networks. Recurrent neural networks (RNN) are widely used in machine learning to solve a variety of sequence learning tasks. In this work we present a train-and-constrain methodology that enables the mapping of machine learned (Elman) RNNs on a substrate of spiking neurons, while being compatible with the capabilities of current and near-future neuromorphic systems. This “train-and-constrain” method consists of first training RNNs using backpropagation through time, then discretizing the weights and finally converting them to spiking RNNs by matching the responses of artificial neurons with those of the spiking neurons. We demonstrate our approach by mapping a natural language processing task (question classification), where we demonstrate the entire mapping process of the recurrent layer of the network on IBM's Neurosynaptic System TrueNorth, a spike-based digital neuromorphic hardware architecture. TrueNorth imposes specific constraints on connectivity, neural and synaptic parameters. To satisfy these constraints, it was necessary to discretize the synaptic weights to 16 levels, discretize the neural activities to 16 levels, and to limit fan-in to 64 inputs. Surprisingly, we find that short synaptic delays are sufficient to implement the dynamic (temporal) aspect of the RNN in the question classification task. Furthermore we observed that the discretization of the neural activities is beneficial to our train-and-constrain approach. The hardware-constrained model achieved 74% accuracy in question classification while using less than 0.025% of the cores on one TrueNorth chip, resulting in an estimated power consumption of ≈ 17μW.

We present an approach to constructing a neuromorphic device that responds to language input by producing neuron spikes in proportion to the strength of the appropriate positive or negative emotional response. Specifically, we perform a fine-grained sentiment analysis task with implementations on two different systems: one using conventional spiking neural network (SNN) simulators and the other one using IBM's Neurosynaptic System TrueNorth. Input words are projected into a high-dimensional semantic space and processed through a fully-connected neural network (FCNN) containing rectified linear units (ReLU) trained via backpropagation. After training, this FCNN is converted to a SNN by substituting the ReLUs with integrate-and-fire neurons. We show that there is practically no performance loss due to conversion to a spiking network on a sentiment analysis test set, i.e. correlations with human annotations differ by less than 0.02 between the original DNN and its spiking equivalent. Additionally, we show that the SNN generated with this technique can be mapped to existing neuromorphic hardware - in our case, the TrueNorth chip. Mapping to the chip involves 4-bit synaptic weight discretization and adjustment of the neuron thresholds. The resulting end-to-end system can take a user input, i.e. a word in a vocabulary of over 300,000 words, and estimate its sentiment on TrueNorth with a power consumption of approximately 50 μW.

This paper describes the hardware and software ecosystem encompassing the brain-inspired TrueNorth processor - a 70mW reconfigurable silicon chip with 1 million neurons, 256 million synapses, and 4096 parallel and distributed neural cores. For systems, we present a scale-out system loosely coupling 16 single-chip boards and a scale-up system tightly integrating 16 chips in a 4 × 4 configuration by exploiting TrueNorth's native tiling. For software, we present an end-to-end ecosystem consisting of a simulator, a programming language, an integrated programming environment, a library of algorithms and applications, firmware, tools for deep learning, a teaching curriculum, and cloud enablement. For the scale-up systems we summarize our approach to physical placement of neural network, to reduce intra- and inter-chip network traffic. The ecosystem is in use at over 30 universities and government/corporate labs. Our platform is a substrate for a spectrum of applications from mobile and embedded computing to cloud and supercomputers.

Deep networks are now able to achieve human-level performance on a broad spectrum of recognition tasks. Independently, neuromorphic computing has now demonstrated unprecedented energy-efficiency through a new chip architecture based on spiking neurons, low precision synapses, and a scalable communication network. Here, we demonstrate that neuromorphic computing, despite its novel architectural primitives, can implement deep convolution networks that i) approach state-of-the-art classification accuracy across 8 standard datasets, encompassing vision and speech, ii) perform inference while preserving the hardware's underlying energy-efficiency and high throughput, running on the aforementioned datasets at between 1200 and 2600 frames per second and using between 25 and 275 mW (effectively > 6000 frames / sec / W) and iii) can be specified and trained using backpropagation with the same ease-of-use as contemporary deep learning. For the first time, the algorithmic power of deep learning can be merged with the efficiency of neuromorphic processors, bringing the promise of embedded, intelligent, brain-inspired computing one step closer.

We demonstrate how to map Local Binary Patterns (LBP), a class of leading feature extractors, onto a neuromorphic processor such as TrueNorth, a silicon expression of a non-von Neumann, low-power, spiking-based, brain-inspired processor. The application is presented in the form of a texture feature extractor that can process 8-bit grayscale video at 30fps. While consuming less than 140mW of power, this neuromorphic implementation provides a rotation and contrast insensitive characterization of texture, with similar accuracy as a standard von Neumann implementation of the same algorithm. The successful mapping of an important vision routine on a neuromorphic architecture is indicative of an alternative paradigm for addressing the von Neumann bottleneck, which is currently placing severe constraints on the processing speed, power consumption, reliability, scalability, programmability and mobility of vision algorithms. This also introduces a new methodology for the design of vision algorithms for power efficient, asynchronous, mobility-targeted applications.