New business models in LLM workloads are reshaping the interconnect landscape. While ongoing challenges such as low latency, high bandwidth, and reliability persist, the emergence of scale-up networking creates new opportunities for performance with orders of magnitude more bandwidth. On the other end of the spectrum, applications are scaling across multiple data centers, trying to harness a million endpoints. The ability of software to deliver performance across the system holistically has become more challenging than ever, pushing the need for co-design. This HotI panel assembles experts from across the hardware and software stack to foster a discussion of these complex, multi-dimensional problems. The panel will explore the trade-offs between closed, and open AI models. Simultaneously, the panel will feature bleeding edge application developers, who will share their experiences leveraging current hardware, articulate their most pressing challenges, and envision the game-changing features that could revolutionize interconnect performance for their applications.
Training and inference of large-scale machine learning models place diverse demands on compute systems, with increasingly higher requirements for processing power, memory capacity, and memory bandwidth. Current compute systems mainly comprise CUs tightly coupled to a fixed amount of high-bandwidth memory (HBM), which needs to be placed in close proximity to the compute die due to constraints related to signal integrity, energy consumption and limited die periphery. This tight coupling limits the system's ability to flexibly adapt to varying compute and memory demands. In this work, we start by exploring the design space of co-packaged photonic I/Os and identifying compute die's shoreline width as a critical architectural constraint. We then present SiPAM, a silicon photonic accelerated memory pooling architecture that integrates dense wavelength division multiplexed (DWDM) photonic I/Os directly along the compute die's shoreline. By replacing local HBMs and conventional electrical networking interfaces with photonic I/Os, we build an unified high-bandwidth photonic communication domain that enables reconfigurable access to both memory and network resources. This architectural design decouples memory and network scaling from compute packaging constraints and allows the system to flexibly adapt to workload demands. We then develop a quantitative optimization methodology to efficiently allocate compute and memory resources based on workloads and parallelism strategies. Our evaluations show that SiPAM scales effectively with model size and I/O bandwidth, improves compute utilization across GPU and memory generations, and achieves up to 3.5×speedup in end-to-end iteration time and up to 5.46× improvement in system efficiency compared to state-of-the-art Nvidia B100 GPU systems.
We demonstrate extremely low energy micro-LED based links, with arrays of about 300 lanes at 3.5Gb/s on a CMOS ASIC and single lane performance at 11.2 Gb/s, all driven NRZ at about 1pJ/bit inclusive of all elements, and with reach of up to 30 meters. The slow frequency roll-off the LEDs allow such high per lane bandwidths with simple equalization. These types of parallel and energy efficient interconnects could have short-term practical applications in scale-up networks in multi-processor clusters.
Emerging computing applications such as Artificial Intelligence (AI) are facing a memory wall with existing on-package memory solutions that are unable to meet the power-efficient bandwidth demands. We propose to enhance UCIe with memory semantics to deliver power-efficient bandwidth and cost-effective on-package memory solutions applicable across the entire computing continuum. We propose approaches by reusing existing LPDDR6 and HBM memory through a logic die that connects to the SoC using UCIe. We also propose an approach where the DRAM die natively supports UCIe instead of the LPDDR6 bus interface. Our approaches result in significantly higher bandwidth density (up to 10x), lower latency (up to 3x), lower power (up to 3x), and lower cost compared to existing HBM4 and LPDDR on-package memory solutions.
The unabated growth in AI workload demands is driving the need for concerted advances in compute, memory, and interconnect performance. As traditional semiconductor scaling slows, high-speed interconnects have emerged as the new scaling engine, enabling the creation of larger logical GPUs by linking many GPUs into a single, low-latency, high-bandwidth compute domain. While initial scale-up fabrics leveraged copper interconnects for their power and cost advantages, the maximum reach of passive electrical interconnects (approximately 1 meter) effectively limits the scale-up domain to within a single rack. The advent of 3D-stacked optics and logic offers a transformative, power-efficient scale-up solution for connecting hundreds of GPU packages (thousands of GPUs) across multiple data center racks. This work explores the design tradeoffs of scale-up technologies and demonstrates how frontier LLMs necessitate novel photonic solutions to achieve aggressive power and performance targets. We model the benefits of 3D CPO (Passage) enabled GPUs and switches within the scale-up domain when training Frontier Mixture of Experts (MoE) models exceeding one trillion parameters. Our results show that the substantial increases in bandwidth and radix enabled by 3D CPO allow for an 8X increase in scale-up capability. This affords new opportunities for multi-dimensional parallelism within the scale-up domain and results in a 2.7X reduction in time-to-train, unlocking unprecedented model scaling.
