RDMA is Turing complete, we just did not know it yet!

It is becoming increasingly popular for distributed systems to exploit network offload to alleviate load on the CPU. Remote Direct Memory Access (RDMA) NICs (RNICs) are one such device, allowing applications to offload remote memory accesses. However, RDMA still requires CPU intervention for complex offloads, beyond simple remote memory access. As such, the offload potential for RNICs is limited and RDMA-based systems usually have to work around such limitations. We present RedN, a principled, practical approach to implementing complex RNIC offloads, without requiring any hardware modifications. Using self-modifying RDMA chains, we lift the existing RDMA verbs interface to a Turing complete set of programming abstractions. We explore what is possible in terms of offload complexity and performance with just a commodity RNIC. Through a key-value store use case study, we show how to integrate complex RNIC offloads into existing applications. RedN can outperform one and two-sided RDMA implementations by up to 3x and 7.8x for key-value get operations and performance isolation, respectively, and provide applications with failure resiliency to OS and process crashes.