Massively Parallel FPGA Hardware for Spike-By-Spike Networks

The increase in computational power by faster computers as well as application specific integrated circuits (ASICs) lead to re-consideration of neuronal networks for pattern recognition and other applications in artificial intelligence (AI). Fueled by this technological progress, large improvements were made, up to the advent of self-driving cars. Often the underlying deep neuronal networks are derived from the many decades old multi-layer preceptron. While inspired by the brain, technical applications lack many key features of the biological original. For example the exchange of information among neurons by electrical pulses (spikes) is missing. On the other hand, approaches using spikes in neuronal networks typically are the more computationally demanding, the more realistic they get. Among the less demanding models, there is a network type called Spike-By-Spike (SbS) network that realizes a compromise between machine learning approaches and biologically realistic models. In this network type computations occur onlyif an incoming spike is detected by a group of neurons. Even though the underlying equations are simple, such a network -- simulated on a normal computer -- is several orders of magnitude slower than multi-layer perceptron based deep network with comparable size and task. One of the main reasons lies in large number of memory accesses for intermediate results during the calculations, which can be reduced if specialized computational pipelines are used. The other main reason is that a commercial CPU has only up to several dozens of cores. Thanks to the communication via spikes, already a small deep network based on SbS type neurons allows a parallelization into thousands of fully independent and simple computational cores. While running a deep SbS network on a normal processor takes a lot of time, it would thrive on an ASIC with many optimized computational SbS cores. In particular, larger networks composed of many such ASICs have the potential to out-compete even the largest presently available supercomputers. In this paper we develop and investigate a framework as well as these computational SbS cores for a network on chip. As long term goal, an implementation with as many as possible SbS cores on an ASIC is envisioned. Here we demonstrate the feasibility of our design on a Xilinx Virtex 6 FPGA while avoiding all types of Xilinx proprietary cores (except block memory) that can not be realized on a custom-designed ASIC. We present memory access optimized circuits for updating the internal variables of the neurons based on incoming spikes as well as for learning the connection9s strength in between neurons. The optimized computational circuits as well as the representation of variables fully exploit the non-negative properties of all data in the SbS network and compare the sizes of the arising circuits for floating and fixed point numbers. In addition we show how to keep the number of components required for the computational cores small by reusing their components for different functions. Furthermore we present the necessary supporting modules required for operating the network in an autonomic fashion on a chip.