How to Train Your Guardian - Evaluating Cyber Security Exercises Using Situation Awareness.

The pressing societal concern for cyber security is highly dependent on the performance of human operators defending critical infrastructure. Cyber security training and Cyber Defence Exercises (CDXs) are increasingly used to improve the skills and awareness of cyber defenders. This paper investigates challenges with measuring and evaluating the impact of CDXs and the potential benefit of introducing Situation Awareness (SA) measurements. First, a systematic literature review identified challenges regarding CDX measures and evaluation. Five main challenges were identified: Defining Wanted Outcomes, Operationalising Human Factors, Choosing Metrics, Data Gathering and Analysis and Assessing Long-term Impact. Second, existing SA measurement methods were reviewed in consideration of their compatibility with the context of CDXs. No currently available SA measurements were found to be compatible. Based on these findings, a discussion is presented focusing on a new proposed outcome of CDXs. If CDXs are used to identify present SA-related challenges, several new opportunities are revealed. A new measurement of SA in CDXs was proposed based on self-rated SA combined with self-reported SA challenges. The proposed method has the potential to identify specific needs for increased domain knowledge and contextual understanding for individuals and teams, identify lacking team communication and coordination and the presence of stress-induced overconfidence in cyber defence responses. This implies that an SA measurement tool should be developed and researched further to validate its benefit for CDX evaluation. We conclude that such a tool might enable the use of CDXs to identify specific training and team development needs of cyber guardians.