Low-latency Communication in RISC-V Clusters

Low-latency inter-node communication is important in HPC clusters. In this work, we design and integrate a low-cost interconnect, capable for low-latency user-level communication with open-source RISC-V processors, obviating the need for bulky and expensive network interface cards connected over the PCI. Our lean network interface is connected next to the Load/Store (LD/ST) stage of the RISC-V processor, which we modify to achieve back-to-back stores for the address range dedicated to the the NI. The primitives that we examine are suitable for many-to-one communication and optimized for small messages, while offering reliable delivery and hardware-level protection using protection domains. We also describe our runtime system that presents the NI to user processes with minimal overheads. Our design achieves sub-microsecond (720 ns) user-level latency for small packet generation and transmission between adjacent FPGA nodes containing Ariane RISC-V soft-cores running at 100 MHz. We also present an analytical latency breakdown including key hardware and software components.