There is a great interest in harnessing the advantages of spiking neural networks. An increasing portion of research is focusing on the deployment of such models. The problem of safe decision making has similarities with classical networks. We apply spiking neural networks to time-series classification tasks where their stateful nature is beneficial. We show that the well-known method of Conformal Prediction (CP) is capable of distinguishing between wrong and correct decisions in this setting similar to but while being less expensive than Evidential Deep Learning and Neural Network Ensembles. In this work we argue that classification uncertainty in time should additionally be considered but is not captured by the length of prediction sets output from CP. Our main contribution addresses the issue that existing CP methods for classification do not consider the aforementioned problem. Our method takes as input the prediction sets which can be output from present conformal prediction and then extends these methods by a smoothed length and combined set algorithm. We apply our method to spiking neural network-based classifiers trained on four different time-series datasets. We show that our method outputs a more suitable uncertainty metric at a given point in time than just the unmodified set length of CP for classification.

Physical reservoir computing (RC) is a beyond von-Neumann computing paradigm that harnesses the dynamical properties of a complex physical system (reservoir) to process information efficiently in tasks such as pattern recognition. This hardware-centered approach drastically reduces training efforts and holds potential for significantly reduced energy consumption operation. Magnetic skyrmions, topological, particle-like spin textures, are considered highly promising candidates for reservoir computing systems due to their non-linear interactions and established mechanisms for low power manipulation combined with thermally excited dynamics. So far spin-based reservoir computing has been used for static detection or has been based on intrinsic magnetization dynamics timescales, that require cumbersome rescaling of typically slower real-world data. Here we harness the power of time-multiplexed skyrmion RC by adjusting the intrinsic timescales of the reservoir to the timescales of real-world temporal patterns: we recognize hand gestures recorded with range-doppler radar on a millisecond timescale and feed the data as a time-dependent voltage excitation directly into our device. We observe the temporal evolution of the skyrmion trajectory using read-out at just one position of the reservoir, which allows for scaling onto the nanometer scale. We find that our hardware solution exhibits competitive performance compared with a state-of-the-art energy-intensive software-based neural network. The key advantage of our hardware approach lies in its capacity to seamlessly integrate data from sensors without the need for temporal conversion, thanks to the time-dependent input and tunable intrinsic timescales of the skyrmion dynamics in the reservoir. This feature enables real-time feeding of the reservoir, opening a multitude of applications.

The potential low-energy feature of the spiking neural network (SNN) engages the attention of the AI community. Only CPU-involved SNN processing inevitably results in an inherently long temporal span in the cases of large models and massive datasets. This study introduces the MAC array, a parallel architecture on each processing element (PE) of SpiNNaker 2, into the computational process of SNN inference. Based on the work of single-core optimization algorithms, we investigate the parallel acceleration algorithms for collaborating with multi-core MAC arrays. The proposed Echelon Reorder model information densification algorithm, along with the adapted multi-core two-stage splitting and authorization deployment strategies, achieves efficient spatio-temporal load balancing and optimization performance. We evaluate the performance by benchmarking a wide range of constructed SNN models to research on the influence degree of different factors. We also benchmark with two actual SNN models (the gesture recognition model of the real-world application and balanced random cortex-like network from neuroscience) on the neuromorphic multi-core hardware SpiNNaker 2. The echelon optimization algorithm with mixed processors realizes 74.28% and 85.78% memory footprint of the original MAC calculation on these two models, respectively. The execution time of echelon algorithms using only MAC or mixed processors accounts for ≤ 24.56% of the serial ARM baseline. Accelerating SNN inference with algorithms in this study is essentially the general sparse matrix-matrix multiplication (SpGEMM) problem. This article explicitly expands the application field of the SpGEMM issue to SNN, developing novel SpGEMM optimization algorithms fitting the SNN feature and MAC array.

Data augmentation is a widely used technique to regularize deep learning models. It is especially famous in computer vision due to its simplicity to apply. Literature suggests numerous ways of transforming images without changing the characteristic semantics. However, for data coming from sensors such as radar these approaches are not applicable leading to data augmentation being not commonly performed. To solve this problem and close the gap we investigate how a Variational Autoencoder (VAE) can be trained on radar data to sample from the learned latent space and use the resulting data to regularize the training of a classifier. We run our experiments on two radar gesture datasets and show that the introduction of generated data can increase generalization. We investigate whether the learned embedded space is sufficient and propose how to sample from the latent space while preserving labels for successful supervised training.

The CPU-based system is widely used for simulating the brain-inspired spiking neural networks (SNN) by taking the benefit of flexibility, while processing high input spiking rates caused by immature coding mechanism costs many CPU cycles, and the introduction of additional information required by serial execution needs the time-consuming pre- and post-neuron matching algorithm. To address these issues, we propose an algorithm set leveraging the multiply-accumulate (MAC) array to accelerate the SNN inference. By rearranging and compressing operands losslessly, we retain the advantage of the MAC array on fast parallel computing, as well as alleviate the ineffective memory occupation and the waste of computing resources, which result from the inherent sparse feature of SNN and reluctant memory alignment from fixed MAC hardware structure. Benchmarking with an SNN radar gesture recognition model, the algorithms jointly optimize 82.71% of the execution time compared to the serial computation on the ARM M4F of the SpiNNaker 2 chip; 49.89% of the memory footprint is reduced contrasted with the unoptimized MAC calculation. This article explicitly expands the application field of the General Sparse Matrix-Matrix Multiplication (SpGEMM) issue to SNN, developing novel SpGEMM optimization algorithms fitting the SNN feature and MAC array.

Spiking neural networks present a new generation of artificial neural networks utilizing binary data representation for large energy, memory, and latency savings due to widespread sparsity. At the same time, they are at the focal point for neuroscientific communities, enabling biologically plausible simulations of systems with a learning capacity. In this work, we present spiking neural networks based on an implementation of a biologically highly plausible neuron model, the Hodgkin-Huxley model. The networks are trained via backpropagation through time using surrogate gradients. They show competitive performances on various neuromorphic datasets while displaying a spiking activity lower by up to two orders of magnitude compared to similar spiking neural networks based on the conventional leaky integrate-and-fire neuron model. Additionally, this work paves the way toward higher biological plausibility in high-performance computational neuroscience by enabling in-depth looks at the underlying dynamics of Hodgkin-Huxley neurons as part of larger operating networks.

Artificial neural networks showed astonishing results in the last decades. However, they tend to consume large amounts of energy which is problematic on edge devices such as wearables where the energy budget is limited. We propose a hybrid neural network of convolutional layers and spiking neural networks that combines the feature extraction capabilities of CNNs with the energy efficiency of SNNs for low-power radar signal processing. This approach is applied to our own radar-based gesture recognition dataset and the publicly available Soli dataset. The hybrid neural network achieves competitive or better accuracies on both datasets in comparison to similarly sized spiking and traditional networks. Experiments on the SpiNNaker 2 system show that they are a little less efficient with noisy input data compared to standard SNNs. The proposed network architecture is a straight forward approach for low-power signal processing of radar data.

Neuromorphic hardware has been emerging in recent years, seeking various applications to explore its uniqueness, limitations, and possibilities. As a representative application and research area, gesture recognition is gaining wider popularity, while the conflict of spiking neural network (SNN) size and available memory of neuromorphic edge-AI can be a thorny issue, which is even intensified by the demand for continuously processing input data stream from the sensor in the real-world scenario since a certain amount of memory is required to ensure that no data loss or overwrite happens. In this paper, an SNN-based real-time radar gesture recognition closed-loop system is proposed, with Infineon's 60 GHz radar continuously capturing motion and the multi-core neuromorphic hardware SpiNNaker 2 serving as the backend to classify gestures with an SNN. A PC is used to preprocess the data and manipulate an actuator. This system requires less than 8 k operation cycles per processor for each radar frame and achieves a classification accuracy of 98.83% with an 8-bit quantized model, with only 134.2 kB memory usage on three processing elements (PEs) and low energy cost. Besides, it performs well in the real-time closed-loop test with 35 ms latency.

Spiking neural networks (SNN) are a promising approach for low-power edge AI (artificial intelligence), especially when run on dedicated neuromorphic hardware. In this work we set up a SNN in TensorFlow, directly train it in a supervised manner with backpropagation through time (BPTT) and surrogate gradients, and compare it with traditional neural networks, based on convolutional neural networks (CNN) and long-short term memory (LSTM) cells, for radar-based hand-gesture recognition. We demonstrate that a small SNN with only 30 hidden leaky integrate-and-fire (LIF) neurons and threshold encoding can achieve an accuracy of 98.1%. With the more complex adaptive LIF neuron, the activity can be reduced by up to 37.9% without significant loss in accuracy. A comparison to traditional LSTM-networks shows the superiority of the SNN in terms of accuracy and computation cost, indicating that they are a considerable alternative to LSTM-based approaches.

This live demo aims at continuously real-time classifying radar gesture signals from the real world with the neuromorphic hardware SpiNNaker 2 prototype to play the game. With the 10 MHz operation frequency on SpiNNaker 2 FPGA, the closed-loop setup realizes around 35 ms delay from PC sending input data to receiving classification output, and there is nearly no feeling of apparent delay when testers are playing the game. The energy cost per frame is 3.29 µJ, and the operation cycle is less than 8 k. Even if our current middleware has not considered balanced work loading among different processing cores, the tightly couple memory usage on the heaviest loaded processing element is less than half of the total 128 kB available memory space based on the directly trained gesture recognition spiking neural network (SNN) model with 2048 input neurons, 5 hidden neurons, and 4 classification outputs.

Spiking neural networks offer a promising approach for low power edge applications, especially when run on neuromorphic hardware. However, there are no well established approaches to setup such networks for real world applications. We demonstrate the use of spiking neural networks on the basis of radar data-based gesture recognition, while taking three different angle-encoding schemes into account, considering a two antenna based angle estimation. The surrogate gradient approach is used for direct training, while achieving a reasonable accuracy on all proposed encoding schemes. This work proposes a baseline approach for spiking networks and the corresponding encoding for the use in radar-based gesture classification.