Real-time strategy games share many aspects with real situations in domains such as battle planning, air traffic control, and emergency response team management which makes them appealing test-beds for Artificial Intelligence (AI) and machine learning. End-user annotations could help to provide supplemental information for learning algorithms, especially when training data is sparse. This paper presents a formative study to uncover how experienced users explain game play in real-time strategy games. We report the results of our analysis of explanations and discuss their characteristics that could support the design of systems for use by experienced real-time strategy game users in specifying or annotating strategy-oriented behavior.

Real-time strategy games, such as Wargus, are examples of complex learning and planning domains that present unique challenges to AI and machine learning. These games usually comprise a large number of states, actions, resources, and decisions that a player needs to take into consideration, while, at the same time, the current game situation is influenced and modified by opponents. With the drive to acquire planning knowledge from ever fewer examples, learning and planning in this complex, dynamic environment is even more challenging. Some headway could be made by providing notations in which an expert can annotate examples to help derive additional knowledge. Previous research in the machine learning and artificial intelligence communities has focused mainly on identifying a language for representing problems and solutions in real-time strategy games (Fern, Yoon, & Givan 2004; Geffner 2004; Khardon 1999; Ponsen & Spronck 2004; W.Aha, Molineaux, & Ponsen 2005). Rather than finding a representation most suitable for machines, we are interested in finding a language or representation most suitable for people to state their intended behavior. Describing behavior can be problematic if there is a significant mismatch between the notation and the user’s conceptualization of their behavior. We seek a natural way for users to express strategic behavior in game-like worlds. To that end, we conducted a formative study with expert players of a real-time strategy game to determine the structure of the language used by the experts to describe strategy. In this paper, we describe a user-centered approach for capturing and analyzing the language of players of the realtime strategy game Wargus, the open source reimplementation of the popular Warcraft II (Blizzard 1995). Our formative study followed a think-aloud design, in which we paired five experts with five novices to help elicit explanations of strategies. The experts were asked to play the game while “thinking aloud” about what they were doing. The novices were instructed to ask the experts about any detail they did not understand. Each expert game-playing session lasted approximately 35 minutes, during which two games were played by the expert. We recorded a video of the game play as well as the interaction between the expert and novice. Afterward, all participants were asked to fill