Design and Experimentation of an Automated Performance Evaluation Testbed for Self-Healing and Self-Adaptive Distributed Software Systems

Evaluating the performance of distributed software systems is very challenging especially in the presence of failures and adaptation. Of particular interest to this paper is self-healing and self-adaptation middleware that detects failures of distributed software systems, analyzes their root causes, devises plans to recover from these failures, and executes these plans. Recovery plans may trigger software architecture adaptations, which may be also initiated by the need to maintain performance and availability goals. This paper focuses on the evaluation and testing of recovery and adaptation frameworks (RAF) for distributed component-based software systems. We present TESS, a testbed for automatically generating distributed software architectures and their corresponding runtime applications, deploying them to the nodes of a cluster, running many different types of experiments involving failures and adaptation, and collecting in a database the values of a variety of failure recovery and adaptation metrics. Queries can then be run against the database to provide a thorough and scientific analysis of the efficiency and/or effectiveness of a RAF. Additionally, this paper presents a case study of the use of TESS for the evaluation of a specific RAF, called DARE, developed by our group.