Cloudweaver: Adaptive and Data-Driven Workload Manager for Generic Clouds

Cloud computing denotes the latest trend in application development for parallel computing on massive data volumes It relies on clouds of servers to handle tasks that used to be managed by an individual server With cloud computing, software vendors can provide business intelligence and data analytic services for Internet scale data sets Many open source projects, such as Hadoop, offer various software components that are essential for building a cloud infrastructure Current Hadoop (and many others) requires users to configure cloud infrastructures via programs and APIs and such configuration is fixed during the runtime In this chapter, we propose a workload manager (WLM), called Cloud Weaver, which provides automated configuration of a cloud infrastructure for runtime execution The workload management is data-driven and can adapt to dynamic nature of operator throughput during different execution phases Cloud Weaver works for a single job and a workload consisting of multiple jobs running concurrently, which aims at maximum throughput using a minimum set of processors