Benchmarking and Characterization of event-based Neuromorphic Hardware

We present the modular framework SNABSuite (Spiking Neural Architecture Benchmark Suite) for “black-box” benchmarking of neuromorphic hardware systems and spiking neural network software simulators. The motivation for having a coherent collection of benchmarks is twofold: first, benchmarks evaluated on different platforms provide measures for direct comparison of performance indicators (e.g. resource efficiency, quality of the result, robustness ...). By using the platforms as they are provided for possible end-users and evaluating selected performance indicators, benchmarks support the decision for or against a system based on use-case requirements. Second, benchmarks may reveal opportunities for effective improvements of a system and can contribute to future development. Systems like the Heidelberg BrainScaleS-project [2], IBM TrueNorth [3], the Manchester SpiNNaker chip [4] or the Intel Loihi platform [5] drive the evolution of neuromorphic hardware implementations, while comparable benchmarks and corresponding measures are still rare. The problem of “comparable” measures can be addressed in two ways: concerning application driven measures like classification accuracy it may be advantageous to implement these algorithms in a highly specialized manner to get the best result