Model-Integrated Queries for the Analysis of Runtime Events: A Controlled Experiment

Models describe a software system on an abstraction level higher than its actual implementation. Recent research results show that bringing models and a running system closer together by establishing traceability links between recorded runtime events and corresponding model elements improves the analysis performance of human observers when assessing the behaviour of the running system. Despite these results, common techniques for analyzing runtime events are rarely integrated into the models that are used for assessing the system behaviour from a high-level perspective. This paper presents a controlled experiment where model-integrated analysis facilities are compared with a more traditional analysis approach based on SQL queries to a system's database in terms of correctness and completion time of analysis tasks. The results show that model-integrated analyses allow analysts to give more correct answers to questions about the system behaviour, but provide no improvement of the time spent for completing the analysis tasks.