Continuous integration for iterative validation of simulated robot models

Simulated environments often provide the first, and are usually the most frequent, test environment for robotic systems, primarily due to their cost and safety advantages. Unfortunately, changing aspects of both, the simulation and the real robot, as well as actuator control algorithms are often not taken into account when relying on simulation results. In this paper we present a continuous integration approach to verify simulated robot models in an integrated and frequent manner, comprising a simulated and a real robot for comparison. The central aspect of our concept is to iteratively assess the fidelity of simulated robot models. In an exemplary case study we distilled a first set of requirements and metrics, which can be used by developers to verify their algorithms and to automatically detect further system changes.