Towards a scalable and energy-efficient resource manager for coupling cluster computing with distributed embedded computing

Microservers (MSs, ARM-based mobile devices) with built-in sensors and network connectivity have become increasingly pervasive and their computational capabilities continue to be improved. Many works present that the heterogeneous clusters, consist of the low-power MSs and high-performance nodes (x86-based servers), can provide competitive performance and energy efficiency. However, they make simple modifications in existing distributed computing systems for adaptation, which have been proven not to fully exploit the various heterogeneous resources. In this paper, we argue that these heterogeneous clusters also call for flexible and efficient computational resource sharing and scheduling. We then present Aries, a platform to support abstracting, sharing and scheduling the cluster resources, scaling from embedded devices to high performance servers, between multiple distributed computing frameworks (Hadoop, Spark, etc.). In Aries, we propose a two-layer scheduling mechanism to enhance the resource utilization of these heterogeneous clusters. Specifically, the resource abstraction layer in Aries is constructed for overall coordination of resources, which provide computation and energy management. A hybrid resource abstraction approach is designed to manage HS and MS resources in fine and coarse granularity separately in this layer to support efficient resource offer based on “resource slot”. And the task schedule layer supports various sophisticated schedulers of existing distributed frameworks and decides how many resources to offer computing frameworks. Furthermore, Aries adopts a novel strategy to support smart switch in three system models for energy-saving effectiveness. We evaluate Aries by a variety of typical data center workloads and datasets, and the result shows that Aries can achieve more efficient utilization of resources when sharing the heterogeneous cluster among diverse frameworks.