Serverless computing is a cost-effective, on-demand paradigm for deploying application workflows, but it faces a key challenge in efficient resource allocation. Platforms must provision resources that satisfy Service Level Objectives (SLOs) while minimizing operational cost. In real workflows, execution time depends on input size, but most approaches are input-agnostic, causing SLO violations for large inputs and overprovisioning for small ones. Recent input-aware schedulers still rely on static averages along the critical path, failing to capture dynamic, input-dependent dataflow across stages. In this work, we propose a Correlation-Aware Proportional Critical Path (C-PCP) method that employs a dynamic, input-aware weighting model derived from online profiling data. A correlation matrix captures statistical dependencies between the input and output sizes of sequential functions, enabling dynamic critical path computation, more accurate sub-SLO allocation, and improved resource profile assignment. Additionally, we develop a dynamic programming based theoretical lower bound to provide a rigorous performance benchmark. For experimental validation, workflow task profiling data is collected from execution traces on a Knative-based testbed deployed over Kubernetes, ensuring realistic modeling of workflow behavior. We evaluate the proposed approach on diverse real-world and synthetic workflows, demonstrating consistent improvements over state-of-the-art methods in both SLO satisfaction and cost efficiency.
Serverless computing has emerged as a promising paradigm for executing scientific workflows characterized by complex task dependencies, data-intensive operations, and high computational demands. However, most existing scheduling approaches assume homogeneous, single-region environments and primarily focus on isolated function execution. These assumptions overlook two critical challenges: (i) the inherent data-parallel nature of workflow tasks, and (ii) the heterogeneity of computing resources across geo-distributed serverless platforms. In this paper, we address these limitations by proposing a novel scheduling framework for geo-distributed serverless environments that explicitly models intra-function data parallelism, heterogeneous abstract resources, and regional concurrency constraints. We formulate a makespan minimization problem in which each function can either execute entirely on a single high-capacity resource or be partitioned across multiple heterogeneous lower-capacity resources, subject to region-specific concurrency limits.To solve this problem, we design a Deep Q-Network (DQN)-based scheduler augmented with two auxiliary heuristics. The first, Critical Workload First, prioritizes high-workload functions through an exhaustive split-deployment search over heterogeneous abstract serverless resources. The second, Load-Aware Heuristic, selects execution regions using a weighted load metric combined with penalty-based resource assignment. We evaluate our approach on five representative scientific workflows BWA, Montage, Inspiral, CyberShake, and SIPHT and using real-world round-trip time measurements from Azure Function deployments across three geo-distributed regions. Experimental results demonstrate that our DQN-based scheduler reduces makespan by up to 29.13% compared to state-of-the-art approaches.
The relentless growth of sizes and traffic volumes in data center networks (DCN) is posing a significant challenge on switching: A single giant switch in a modern flattened-topology DCN needs to direct terabits of traffic per second to hundreds of top-of-rack switches in a low latency of just a few milliseconds. The vast majority of existing switching schedulers have yet to address this challenge: Traffic-oblivious RODCN (reconfigurable optical DCN) schedulers do not have high throughput utilization, whereas on-demand hybrid (circuit and packet) switching schedulers are too computationally expensive. To this end, we propose an efficient and performant hybrid switching scheduler, named QPS-Fit, which computes high-quality on-demand schedules in a fully parallelizable way. Our algorithm simultaneously fits many transmissions into the schedule, all by parallel "dirt-cheap" request-grant message exchanges between input and output ports. According to our simulation, QPS-Fit achieves similar throughput utilization as BFF, the state-of-the-art on-demand hybrid switching scheduler, while being nearly 27 x faster when running on 96 threads.
The rapid explosion of Artificial Intelligence (AI) workloads utilizing a growing number of accelerators has placed unprecedented demand on the network. These workloads typically leverage Remote Direct Memory Access (RDMA) and require a high-performance network fabric. While many purposebuilt cloud networking solutions can provide high performance, efficiently utilizing these costly infrastructures require a fabric that is multi-tenant for ease of consumption. Furthermore, the fabric must be resilient to faults and for operational manageability. Resilient cloud networks typically employ mature Ethernet segmentation techniques over Clos topologies with Equal Cost Multi Path (ECMP) routing. ECMP hashes flows to paths, which in case of collisions can significantly degrade performance for large RDMA over Converged Ethernet (RoCE) flows. To mitigate ECMP penalties, we evaluate routing strategies with varying levels of operational complexity. We explore load balancing and path pinning solutions that leverage non-proprietary, mature technologies over commodity Ethernet. Our evaluation follows a three-fold strategy, focusing on the key dimensions of performance, resiliency, and operational complexity. By applying this methodology to representative implementations, we highlight the trade-offs. While all techniques are resilient, path pinning-based solutions excel at performance but introduce greater complexity. Specifically, path pinning achieves up to 1.6x improvement over ECMP for RoCE test traffic and up to 2.5x for NCCL AllReduce. These results validate the promising performance benefits of path pinning and highlight the need to explore less complex implementations for broader adoption. Our methodology can be used to rigorously evaluate future implementations in support of AI network design.
Managing diverse data formats and improving query processing efficiency at cloud service centers is crucial in the era of interconnected heterogeneous devices. Lakehouse effectively manages heterogeneous data but faces challenges in maintaining SQL query performance and data independence in large-scale analyses. Precomputation, which stores intermediate results in advance, reduces query latency by a space-for-time trade-off. This paper addresses the challenges of matching and rewriting results in lakehouse environments. By combining pre-computation with the low-cost and dynamically scalable storage characteristics of cloud storage, we introduce an optimized lakehouse architecture with an in-memory index and dynamic task scheduling to enhance OLAP performance while balancing data center resources. Experiments demonstrate improvements in query performance ranging from 6.1 % to 64.9% over native lakehouse architectures and reductions in resource occupation from the original 75.6% to 17.3% with preloading and 25.0% with dynamic scheduling.
