ReWorDs 2022 Keynote: Towards Orchestrating Distributed & Data-Intensive Workflows

Scientific exploration and hypothesis generation is increasingly dependent on the convergence of scientific modeling, data analytics, and machine learning. The result is data-intensive workflows that are composed of multiple stages of computation and communication between distributed and heterogeneous computing resources. The key bottleneck is frequently non-scalable data management, resulting in problematic data volumes and velocities. This talk will discuss our efforts characterizing, modeling, analyzing, and accelerating the performance of these emerging workloads. It will also sketch challenges of enabling workflows, written in a variety of languages, that have production-like performance in massively distributed environments with multiple computing paradigms.