Inter-node Communication in Multi-FPGA-based Computational Neuroscience Simulators: PHY-to-Link

High-performance simulation platforms are integral to advancing research in computational neuroscience. Recently, the development of multi-FPGA platforms for such simulations has gained prominence due to the inherent flexibility of FPGAs, allowing them to adapt to the evolving demands of computational neuroscience. These platforms have reached sub-realtime simulation capabilities for medium-sized models with realistic biological considerations. Despite leveraging communication architectures with minimized network diameter and traveling distance, further acceleration is constrained by the communication architecture, specifically the communication latency. This study addresses the latency bottleneck associated with inter-FPGA communication in computational neuroscience simulators. The serial interface and the accompanying communication protocol significantly contribute to this latency. The existing solutions optimize the protocol without considering their design impact on the interface latency, which must be considered when transferring a vast number of small packets - the typical traffic load in computational neuroscience simulators. To tackle this, we introduce a customized ultra-low latency serial link utilizing FPGA-embedded multi-gigabit transceivers. This design offers the flexibility to adapt to various application demands while minimizing latency by co-optimizing the physical and link layers. Experimental results on the Xilinx U200 platform demonstrate that a 128-bit packet can be transmitted from a router to a neighboring router over a 15.625 Gbps link in 34 ns. Our analyses show that employing the proposed customized link can lead to a 72-fold speedup in the neuroAIx platform when simulating the microcircuit model.