Spiking neural networks (SNNs) are increasingly applied to event-based data generated by event cameras due to their asynchronous and sparse properties. Event cameras can inherently respond to the changes in the scene, which is a quite desirable property for action recognition tasks. However, existing works of SNNs for event-based action recognition are still limited. To capture the rich dynamics embedded in event streams, we propose the two-stream SNN that consists of spatial spiking stream and motion spiking stream to address event-based action recognition. To effectively build the two-stream SNN, we present a motion feature aggregation strategy and an attention-based two-stream fusion method. The motion feature aggregation strategy accumulates motion information and groups it into distinct channels for input into the SNN, which can alleviate the dilemma of information loss caused by compact representation. The attention-based two-stream fusion method can fuse the spatial and motion features effectively using the channel-wise attention mechanism, which helps our network to achieve better integration of two-stream information. Extensive experimental results on three event-based action recognition datasets show our proposed two-stream SNN achieves competitive performance with much fewer trainable parameters, which demonstrates the effectiveness of our work in event-based action recognition tasks.

DThe advantages of high temporal resolution and high dynamic range provided by event cameras are particularly suitable for moving object detection, especially in scenarios with motion blur and extreme lighting conditions. Current popular methods predominantly focus on designing powerful network architectures to extract event features, often neglecting the rationality of event representation design which has been proven to impact significantly on downstream tasks. In particular, current event representations typically rely on fixed hyperparameters, without considering variations in relative motion speed, a key factor in motion -rich scenes captured by event cameras. To tackle this challenge, we propose a gradient -based scaled Timesurface (STS) motivated by the observation of the relationship between motion speeds and the gradient strength, which adaptively rescales the decay factor at different spatial positions. Additionally, we propose a dataset called Rotatel)igit, which is the first event dataset featuring clear motion level annotations to our best knowledge. Proposed STS method is verified using Spiking Neural Network (SNNs) due to the sharing asynchronous and sparse properties with event camera. Experimental results on RotateDigit and Geld_ show the performance improvement achieved by STS, which validates the rationality and effectiveness of our work.

Event cameras, with their high dynamic range and temporal resolution, are ideally suited for object detection, especially under scenarios with motion blur and challenging lighting conditions. However, while most existing approaches prioritize optimizing spatiotemporal representations with advanced detection backbones and early aggregation functions, the crucial issue of adaptive event sampling remains largely unaddressed. Spiking Neural Networks (SNNs), which operate on an event-driven paradigm through sparse spike communication, emerge as a natural fit for addressing this challenge. In this study, we discover that the neural dynamics of spiking neurons align closely with the behavior of an ideal temporal event sampler. Motivated by this insight, we propose a novel adaptive sampling module that leverages recurrent convolutional SNNs enhanced with temporal memory, facilitating a fully end-to-end learnable framework for event-based detection. Additionally, we introduce Residual Potential Dropout (RPD) and Spike-Aware Training (SAT) to regulate potential distribution and address performance degradation encountered in spike-based sampling modules. Through rigorous testing on neuromorphic datasets for event-based detection, our approach demonstrably surpasses existing state-of-the-art spike-based methods, achieving superior performance with significantly fewer parameters and time steps. For instance, our method achieves a 4.4\% mAP improvement on the Gen1 dataset, while requiring 38\% fewer parameters and three time steps. Moreover, the applicability and effectiveness of our adaptive sampling methodology extend beyond SNNs, as demonstrated through further validation on conventional non-spiking detection models.

[This corrects the article DOI: 10.3389/fnins.2023.1123698.].

Event cameras are asynchronous and neuromorphically inspired visual sensors, which have shown great potential in object tracking because they can easily detect moving objects. Since event cameras output discrete events, they are inherently suitable to coordinate with Spiking Neural Network (SNN), which has a unique event-driven computation characteristic and energy-efficient computing. In this paper, we tackle the problem of event-based object tracking by a novel architecture with a discriminatively trained SNN, called the Spiking Convolutional Tracking Network (SCTN). Taking a segment of events as input, SCTN not only better exploits implicit associations among events rather than event-wise processing, but also fully utilizes precise temporal information and maintains the sparse representation in segments instead of frames. To make SCTN more suitable for object tracking, we propose a new loss function that introduces an exponential Intersection over Union (IoU) in the voltage domain. To the best of our knowledge, this is the first tracking network directly trained with SNN. Besides, we present a new event-based tracking dataset, dubbed DVSOT21. In contrast to other competing trackers, experimental results on DVSOT21 demonstrate that our method achieves competitive performance with very low energy consumption compared to ANN based trackers with very low energy consumption compared to ANN based trackers. With lower energy consumption, tracking on neuromorphic hardware will reveal its advantage.