Foxhound: Server-Grade Observability for Network-Augmented Applications

There is a growing move to offload functionality, e.g., TCP or key-value stores, into programmable networks - either on SmartNICs or programmable switches. While offloading promises significant performance boosts, these programmable devices often provide little visibility into their performance. Moreover, many existing tools for analyzing and debugging performance problems, e.g., distributed tracing, do not extend into these devices. Motivated by this lack of visibility, we present the design and implementation of an observability framework called Foxhound, which introduces a co-designed query language, compiler, and storage abstraction layer for expressing, capturing and analyzing distributed traces and their performance data across an infrastructure comprising servers and programmable data planes. While general, Foxhound's query language offers optimized constructs which can circumvent limitations of programmable devices by pushing down operations to hardware. We have evaluated Foxhound using a Tofino switch and a large scale simulator. Our evaluations show that our storage layer can support common tracing tasks and detect associated problems at scale.