We propose remote direct code execution (RDX), which elevates the power of RDMA from memory access to code execution. We target runtime extension frameworks such as Wasm filters, BPF programs, and UDF functions, where RDX enables an agentless architecture that unlocks capabilities such as fast extension injection, update consistency guarantees, and minimal resource contention. We outline the roadmap for RDX around a new CodeFlow abstraction, encompassing programming remote extensions, exposing management stubs, remotely validating and JIT compiling code, seamlessly linking code to local context, managing remote extension state, and synchronizing code to targets. The case studies and initial results demonstrate the feasibility of RDX and its potential to spark the next wave of RDMA innovations.
Verifiable network telemetry is crucial for ensuring transparency and trust in network measurements. However, telemetry logs (e.g., NetFlow records) often contain sensitive data, making public verification challenging. Recent work has attempted to address this problem using Trusted Execution Environments (TEEs), such as Intel SGX, to provide confidentiality and integrity guarantees. However, TEEs are known to suffer from complex deployment requirements and limited scalability. In this paper, we introduce a software-based approach utilizing the latest advances in Zero-knowledge Proofs (ZKPs) to enable verifiable network telemetry without revealing the underlying sensitive logs or relying on special-purpose hardware. Our system employs a general-purpose ZKP virtual machine (RISC Zero) to generate cryptographic proofs over NetFlow data, enabling operators to securely attest to network flow metrics. Our preliminary results indicate that our ZKP-based design offers a viable path toward overcoming deployment and scalability limitations inherent in the solutions that require special-purpose hardware.
This paper systematically studies the cost impact of distributed in-memory caches on datacenter services. While memory used for these caches is often perceived to be expensive, we find that the resulting CPU savings from these in-memory caches far outweigh the cost of added memory. In fact, across a variety of both synthetic and production workloads, we find that adding distributed in-memory caches can lower total operating costs by 3 - 4x, even without considering their latency benefits. These cost savings can vary significantly across various architectures, such as storage layer caches, remote lookaside caches, and in-memory linked caches. We additionally evaluate cost for two emerging scenarios: caching rich application objects and strongly consistent cache. For the former, we find that caching application objects provides outsized benefits compared to their denormalized, key-value-style variants, up to 8x compared to reading from storage. For the latter, we observe that even a minimal version check for consistency can eliminate most of the cost benefits, calling for new designs for cost-effective consistent cache.
Emerging chiplet-based server platforms and the resulting server chiplet networking present a fundamental shift in the (intra-)host network. Unlike conventional monolithic servers, compute chiplets, I/O chiplets, off-chip memory, and peripheral devices communicate through a collection of heterogeneous interconnects and links, formalizing a new server chiplet network substrate that has not been explored before. This paper makes an initial step by characterizing two generations of AMD EPYC chiplet servers, identifying four communication idiosyncrasies, and summarizing the design implications. We outline some future directions under server chiplet networking and discuss how to build next-generation server systems and applications.
While ML can greatly aid network management, it often makes glaring mistakes that contradict common sense or domain-specific constraints, undermining its trustworthiness and hindering adoption. To address this mismatch, this paper advocates for enforcing logic during ML inference (or Just-In-Time), rather than during training or post-inference in prior work. We find that this approach offers correctness guarantees without sacrificing statistical fidelity, thereby maximizing the benefits of both ML and formal reasoning. To achieve Just-In-Time Logic Enforcement, we interleave an SMT solver into the language model's inference process, which guides generation step by step to enforce domain-specific rules. Our proof-of-concept implementation, LeJIT, turns a generic GPT-2 model at inference time into either a synthetic data generator or a telemetry imputer by applying different sets of logic rules and performs on par with task-specific SOTA systems. LeJit paves the way for a networking foundation model networking that can be repurposed through logic rules, instead of costly retraining or fine-tuning.
Cloud systems constantly experience changes. Unfortunately, these changes often introduce regression failures, breaking the same features or functionalities repeatedly. Such failures disrupt cloud availability and waste developers' efforts in re-investigating similar incidents. In this position paper, we argue that regression failures can be effectively prevented by enforcing low-level semantics, a new class of intermediate rules empirically inferred from past incidents, yet capable of offering partial correctness guarantees. Our experience shows that such rules are valuable to strengthen system correctness guarantees and expose new bugs.
We propose moving from Software Defined Networks (SDN) to Software Managed Networks (SMN) where all information for managing the life cycle of a network (from deployment to operations to upgrades), across all layers (from Layer 1 through 7) is stored in a central repository. Crucially, a SMN also has a generalized control plane that, unlike SDN, controls all aspects of the cloud including traffic management (e.g., capacity planning) and reliability (e.g., incident routing) at both short (minutes) and large (years) time scales. Just as SDN allows better routing, a SMN improves visibility and enables cross-layer optimizations for faster response to failures and better network planning and operations. Implemented naively, SMN for planetary scale networks requires orders of magnitude larger and more heterogeneous data (e.g., alerts, logs) than SDN. We address this using coarsening - mapping complex data to a more compact abstract representation that has approximately the same effect, and is more scalable, maintainable, and learnable. We show examples including Coarse Bandwidth Logs for capacity planning and Coarse Dependency Graphs for incident routing. Coarse Dependency Graphs improve an incident routing metric from 45% to 78% while for a distributed approach like Scouts the same metric was 22%. We end by discussing how to realize SMN, and suggest cross-layer optimizations and coarsenings for other operational and planning problems in networks.
The goal of this paper is to apply two groundbreaking scaling transformations from the computing packaging industry to internet routers: heterogeneous integration of High-Bandwidth Memories (HBMs) and chiplets, as well as inpackage optics. We propose a novel internet router architecture that leverages these technologies to realize a petabit/sec router in a single integrated package. We first introduce a new Split-Parallel Switch (SPS) architecture that spatially splits (without processing) the incoming fibers and distributes them across smaller independent switches. Then, we design these smaller switches as novel shared-memory HBM switches. We show how a Parallel Frame Interleaving (PFI) algorithm packs traffic into frames and accesses the HBM banks in a cyclical staggered interleaved way to reach HBM peak data rates. We further explain why these new technologies represent a paradigm shift in the design of future internet routers. Finally, we highlight that power consumption may constitute the main scaling bottleneck.
Industrial networks are undergoing a radical shift from closed, static OT environments towards open networks that integrate IT and OT. This shift applies IT operation principles to OT environments, such as virtualizing Programmable Logic Controllers and using Artificial Intelligence to increase production and process efficiency. While there is a huge effort to integrate IT principles, this paper demonstrates that IT/OT convergence remains an underexplored area of research, leaving out critical research opportunities for future networking systems. We identify three core challenges: timing constraints, service availability, and changing network traffic characteristics. For each challenge, we provide a concrete use case that demonstrates early findings and opens up new avenues for research within SIGCOMM.
Modern ML workloads increasingly rely on direct communication between host devices-such as GPUs, NVMe SSDs, and DRAM-spanning intra-host and inter-host networks. However, today's intra-host network lacks hardware-level primitives for routing across heterogeneous interconnects, hindering efficient use of alternative paths and leading to suboptimal performance under failures or congestion. Furthermore, the inter-host network treats the NIC as the endpoint, with intra-host interconnects like PCIe running oblivious to inter-host network protocols. This prevents leveraging multiple paths for communication between host devices across different servers. To address these limitations, we propose expanding the datacenter network layer to encompass the intra-host network, making intra-host devices first-class network endpoints. Our scheme envisions hardware-level routing and forwarding across multiple intra-host interconnects and makes intra-host devices visible to the inter-host network. This unified approach provides a principled foundation for robust, efficient peer-to-peer communication between storage and compute hardware devices in AI datacenters.
Modern Internet services generate massive volumes of observability data, yet identifying opportunities for business performance improvements remains elusive. In many cases, such insights manifest only within sub-populations defined by derived attributes that cannot be predefined, might evolve over time, and often cannot be exhaustively enumerated ahead of time. Unfortunately, existing commercial and research systems fall short in one or more aspects of generating such improvement opportunities: expressiveness, automation, and scalability. We present a vision for automatically surfacing opportunities for improvements to tackle these seemingly conflicting and intractable requirements. We highlight the early promise from a proof-of-concept system, showing evaluation on three real-world services and discuss open challenges for future work.
Networking research often abstracts away the people who build, operate, and experience the Internet. Yet, human decisions shape what gets measured, which problems are prioritized, and how solutions are deployed. This paper argues that such human influence is foundational and deserves methodological attention. To do so, we discuss three well-known qualitative methods and approaches: participatory action research, ethnographic methods, and positionality as concrete ways of engaging with the social and operational realities that underlie technical systems. These approaches formalize processes that are commonly implicit in networking research, help surface questions that cannot be answered with traces alone, and make space for voices often left out of the research pipeline or are inadvertently concealed due to the lack of formal procedures to include them in our research methods. Ultimately, we argue for a broader understanding of legitimate, and sometimes informal, contributions to networking research—one that better reflects the human element of how the Internet is structured and is experienced.
Despite decades of investment and regulatory e.orts, Africa's Internet ecosystem still relies heavily on infrastructure far outside the continent, routing tra.c through Europe and outsourcing critical services like DNS resolution. This dependence on foreign infrastructure exacerbates the impact of subsea cable cuts and exposes a deeper problem. That is, the continent's connectivity fabric remains externally dependent and structurally fragile. Existing measurement tools fail to illuminate this reality, either by missing key components or o.ering insu.cient visibility. We argue for a rethinking of how we monitor and support Africa's Internet infrastructure. Our vision is a purpose-built testbed that combines crowd-sourced vantage points with intentional, context-aware targeting to better capture the unique Internet ecosystem of the continent. Rather than simply retro.tting global solutions, we propose building with the realities of Africa's ecosystem in mind.
Privacy policies dictate how systems handle user data, yet engineers struggle to verify compliance because policies use intentionally vague legal language. Current automated analyzers extract data practices using NLP but fail when policies say things like "share data for legitimate purposes" - terms that have no computational definition. This mismatch between legal flexibility and formal verification creates a fundamental barrier to automated compliance checking. We identify four systematic challenges: vague terms, evolving terminology, exception patterns that appear contradictory, and external legal dependencies. We propose an approach that preserves this ambiguity, where we use LLMs to extract structured parameters and convert them to first-order logic while keeping vague conditions as explicit placeholders for human interpretation. Our system can extract hundreds of data practices and reveals hidden complexities in TikTok and Meta policies, though the resulting formulas remain too complex for SMT solvers. This demonstrates the promise and fundamental limits of formalizing the legal text.
Existing frameworks for managing distributed systems hard-code scheduling policies and their implementations (e.g., centralized vs. decentralized), limiting customization and hurting performance across diverse applications and workloads. We argue for an adaptive scheduling approach, where developers express policies in a high-level, framework-agnostic DSL, and a compiler generates optimized implementations based on policy semantics, workload characteristics, and execution environments. We demonstrate that our compilerguided approach can significantly improve both scheduling quality and performance.
AI training now floods datacenter fabrics with thousands of simultaneous collectives, yet most frameworks still move data the hard way: O(N) unicasts for a group of N processors. Classic multicast could slash those bytes but has long been deemed unscalable: computing an optimal tree in an asymmetric Clos is NP-hard and group-specific rules quickly exhaust switch TCAM. We contend that both obstacles disappear once we embrace two mundane facts of today's AI deployments. (i) Layer regularity. Even after link failures, Clos paths rise and fall cleanly through layers. We propose a novel layer-peeling heuristic that exploits this to build near-optimal Steiner trees in polynomial time. (ii) Job locality. GPU schedulers binpack tasks into a few racks, enabling us to design a power-of-two prefix method through existing datacenter switch operations. Pre-installing the k - 1 prefixes per pod, which shrinks state from exponential to linear and adds less than 8 B per packet. In a 64-ary fat-tree (65,536 hosts) our prototype uses just 63 rules, down from four billion-and performs within 1.4% of the Steiner optimum. We do not claim a finished system; rather, we argue that multicast is finally within reach for trillion-parameter models and invite the community to revisit the assumption that "multicast simply doesn't scale."
While encryption has become ubiquitous across the Internet, there is growing concern that website fingerprinting and other traffic analysis attacks could undermine the confidentiality guarantees encryption is meant to provide. Over the past decade, these attacks have become increasingly effective, highlighting the urgent need to deploy traffic obfuscation countermeasures. Although defenses have already been proposed in the literature, they remain inefficient partly because they are implemented at the application-level, which limits their control over packet sequences. This paper advocates for integrating packet sequence obfuscation support directly into host network stacks, where the fine-grained packet operations that defenses require can be effectively enforced.
Network hardware faults are inevitable in massive scale-out ML training clusters. Networks in such systems are inherently designed for resiliency, routing around faulty components as long as a fault is detected. Unfortunately, some silent faults evade detection. Notably, the effects of silent faults are amplified in modern production networks that deploy per-packet load balancing, because packets of a single flow traverse many network paths, making such faults particularly hard to localize. We present FlowPulse, the first system for rapid, low-overhead detection of silent network faults in per-packet spraying networks. Our key insight is that distributed trainingworkloads induce predictable traffic patterns in the switch ports we refer to as a temporal symmetry. This symmetry emerges even in the presence of known faults, and can be modeled analytically or learned from the traffic. FlowPulse detects new network faults of training tasks by identifying subtle deviations from the expected temporal symmetry on each switch during collective communications, all without any inter-switch coordination or probing overheads. Our preliminary results show that FlowPulse is effective in detecting silent faults in a variety of realistic settings, topologies and fault patterns. For example, FlowPulse identifies a single faulty link with 1.5% corruption rate by checking temporal symmetry in a full two-level fat tree topology with 32 leaf switches while performing Ring-AllReduce on all nodes.
In recent years, networking hardware development has primarily focused on speed rather than power efficiency. By contrast, computing hardware has received a lot more attention given its dominant power footprint, especially in machine-learning (ML) data centers. With faster networks, we spend less time communicating and get more useful work out of the (increasingly expensive) computing hardware. But, the faster the network, the more time it idles and the worse its energy efficiency, which is magnified by the notorious lack of power proportionality of networking equipment. In this paper, we analyze the network power footprint in a production ML cluster and find that it accounts for a still sizeable fraction of the total (12%) and that, by improving network power proportionality to match that of the compute, one could save close to 9% of the overall cluster energy demand. We argue that this potential is worth investigating and discuss opportunities and challenges to address power proportionality in networking hardware, which we invite the networking research community to tackle.
Creating and maintaining cloud infrastructure via "DevOps programs" is essential to using the cloud. However, developing and testing the DevOps programs requires resource provisioning in the cloud, which is time-consuming and costly. Cloud emulators seek to enable high velocity development by emulating cloud-level APIs to DevOps programs, enabling frictionless testing locally without going through the cloud. However, developing these emulators today is tedious and error-prone: engineers need to digest extensive documentation, and hand-craft emulation logic for each service and service interactions. We make a case for a fundamentally different approach: to "learn" emulation logic from cloud documentation via automated code synthesis. We observe that this task is particularly amenable to AI automation, and that we can constrain the code generation using principled abstractions for accurate synthesis at scale. We report our preliminary findings and discuss new opportunities that our approach will enable. check
