Frustration is a natural part of learning in AIED systems but remains relatively poorly understood. In particular, it remains unclear how students' perceptions about the learning activity drive their experience of frustration and their subsequent choices during learning. In this paper, we adopt a mixed-methods approach, using automated detectors of affect to signal classroom researchers to interview a specific student at a specific time. We hand-code the interviews using grounded theory, then distill particularly common associations between interview codes and affective patterns. We find common patterns involving student perceptions of difficulty, system helpfulness, and strategic behavior, and study them in greater depth. We find, for instance, that the experience of difficulty produces shifts from engaged concentration to frustration that lead students to adopt a variety of problem-solving strategies. We conclude with thoughts on both how this can influence the future design of AIED systems, and the broader potential uses of data mining-driven interviews in AIED research and development.

Education research has explored the role of students' affective states in learning, but some evidence suggests that existing models may not fully capture the meaning or frequency of how students transition between different states. In this study we examine the patterns of educationally-relevant affective states within the context of Betty's Brain, an open-ended, computer-based learning system used to teach complex scientific processes. We examine three types of affective transitions based on similarity with the theorized D'Mello and Graesser model, transition between two affective states, and the sustained instances of certain states. We correlate of the frequency of these patterns with learning outcomes and our findings suggest that boredom is a powerful indicator of students' knowledge, but not necessarily indicative of learning. We discuss our findings within the context of both research and theory on affect dynamics and the implications for pedagogical and system design.

This paper investigates the interactions between learners’ cognitive strategies and affective states; both important components of self-regulated learning (SRL) processes that influence student learning. We study cognitive-affective relationships in high versus low performing students as they worked on a model building task to teach their agent in Betty’s Brain, an open-ended science learning environment. Our initial results allow for some interesting discussions, but they also emphasize the need for fine grained affective data to match up against cognitive states to determine how they influence performance or vice versa. Introduction Self regulated learning (SRL) involves the temporal deployment of cognitive, affective, metacognitive, and motivational processes (Azevedo et al., 2012). SRL processes are important for successful science learning, especially in open-ended learning environments (OELEs). Related work in this area show that affect can impact cognitive behaviors like decision-making, information processing, and reasoning (Forgas et al., 2006). Previous results highlight the importance of measuring and studying the interactions of affect with students’ learning performance and behaviors. These findings help us understand that detecting and alleviating negative affect leads to better learning. This paper analyzes the interactions between components of SRL, primarily cognitive strategies and affective states, in the Betty’s Brain OELE (Leelawong and Biswas, 2008). Methods We ran a study with 87 sixth-grade students in an urban public school in Nashville, TN, USA, who worked on modeling the causes and effects of climate change in Betty’s Brain over four days. Students’ interactions in Betty’s Brain were logged. Two trained human observers (Cohen’s kappa, κκ > 0.8) collected data on students’ affective states (bored, confused, delight, engaged concentration, frustrated, other), as well as their task behaviors (ontask, off-task, on-task conversation, other) following the Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP) for field observations (Ocumpaugh, Baker, & Rodrigo, 2015). We used differential sequence mining to identify frequent cognitive strategies that differed between high and low performing students (Kinnebrew et al., 2013). We then computed the differences in affect observed for high versus low performers, and thereby studied the relationships between cognitive strategies and on-task affective states. Results and discussions We applied the measure of students’ in-system performance, their map scores (mmmmmmmmmmmm vvmmvvvvmm = 11), to divide all the students (mm = 87) into High (Hi) and Low (Lo) performers. Students with a map score greater than 11 were labeled “Hi” (mm = 44) and those with a map score less than 10 were labeled “Lo” (mm = 39). Data for students at the median value (mm = 4) was discarded to maintain the distinction between the two groups. Next, we ran a differential sequential pattern mining algorithm (Kinnebrew et al., 2013) to identify the differentially frequent cognitive patterns between Hi and Lo performers. Table 1 presents 3 frequent cognitive strategies sorted by the difference in the two groups’ instance support values. The descriptors “-EFF/-INEFF” and “-SUP/-UNSUP” attached to causal link edit actions show whether the actions were effectiveineffective (led to an increasedecrease in the map score) and supportedunsupported (relatedunrelated to previous actions), respectively. Table 1: Differentially frequent patterns for High vs Low performing students Pattern i-Frequency Diff (Hi − Lo) s-Frequent Group pp value READ  LINKADD-EFF-SUP  QUIZ 2.1 Hi < 0.001 QUIZ  READ 4.35 BOTH < 0.001 READ  CONV-REQ −0.47 Lo 0.02 ICLS 2018 Proceedings 1691 © ISLS We analyzed the data collected on affective states of the students by group (Hi and Lo performers). We found that the amount of time spent being bored was significantly higher for the Low performers (pp = 0.002) when the learners were in on-task mode. On-task behavior was also naturally marked by significantly higher level of delight (pp = 0.015) for the Hi group. Next, we studied the affective states during use of the cognitive strategies from Table 1 between Hi and Lo performing students. The results of these cognitive-affective interactions are illustrated in Figure 1. QuizRead, a frequently used pattern by all students, was linked to delight or frustration for the Hi performers, versus boredom and confusion for the Lo performers. A possible interpretation of this result is that, seeing good quiz results or improvement in the results produces delight, whereas negative quiz results produce frustration in the Hi performers. Confusion and boredom in Lo performers can be attributed to not being able to analyze the quiz results to find the information to read. However, frustration in high performers caused by the lack of immediate success did not seem to affect their desire to persist (unlike the low performers who got bored and gave up). Instead, frustration seemed to be channeled into reading activities that helped them find the information that they needed to correct their maps, and, therefore, they got past their frustrated state. ReadConv-Req is a differentially frequent pattern for Low performers. It includes a period of reading resource pages, but not understanding the content (or finding the desired information), therefore, seeking help from the Mentor. This is good SRL behavior, however, the Lo group showed a lot of boredom during this period (implying their lack of success led them to disengage from productive work on system), while the Hi performers primarily displayed confusion or frustration (unable to understand content, therefore, confusion or frustration, but they seemed to follow up by seeking help in a productive way). Read LinkAdd-Eff-SupQuiz is differentially frequent in High performers. Here, the student read and converted the information they read into correct causal links, and then took a quiz to verify if their added link was correct. For Lo performers, who performed this strategy rarely, no affect state was recorded (hence, their affect states are not included in Figure 1). Hi performers displayed significant levels of delight during this period, indicating that they were happy because they updated the map correctly based on what they read in the resources, and the quiz results confirmed that they had added correct link(s). Figure 1. Hi vs Lo students Emotions (on-task) for four different behavioral strategies. Conclusions The findings in this paper show that learners’ affective states in an open-ended learning environment are linked to their cognitive strategies and performance in the system. As a broader research goal, we believe that establishing cause-effect relations between cognitive and affective processes will help us construct more complete models to gain an understanding of the relations between cognition, metacognition, and affect in SRL. References Azevedo, R., Behnagh, R. F., Duffy, M., Harley, J. M., Trevors, G. (2012). Metacognition and SRL in studentcentered learning environments. Theoretical foundations of learning environments, Ch. 7. Forgas, J., Wyland, C., & Lahan, S. (2006). Hearts and Minds: An Introduction to the Role of Affect in Social Cognition and Behavior. Affect in Social Thinking and Behavior, Psychology Press, 3-18. Kinnebrew, J.S., Loretz, K.M., & Biswas, G. (2013). A Contextualized, Differential Sequence Mining Method to Derive Students' Learning Behavior Patterns. Journal of Educational Data Mining, 5(1), 190-219. Leelawong, K., & Biswas, Gautam. (2008). Designing learning by teaching agents: the Betty’s Brain system. International Journal of Artificial Intelligence in Education, 18(3), 181–208. IOS Press. Ocumpaugh, J., Baker, R. S., & Rodrigo, M. M. T. (2015). Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP) 2.0 technical and training manual. Technical Report. New York, NY: Teachers College, Columbia University. Manila, Philippines: Ateneo Laboratory for the Learning Sciences. Acknowledgments This research was supported by NSF Award #1561676. ICLS 2018 Proceedings 1692 © ISLS

The past few years have seen a surge of interest in deep neural networks. The wide application of deep learning in other domains such as image classification has driven considerable recent interest and efforts in applying these methods in educational domains. However, there is still limited research comparing the predictive power of the deep learning approach with the traditional feature engineering approach for common student modeling problems such as sensor-free affect detection. This paper aims to address this gap by presenting a thorough comparison of several deep neural network approaches with a traditional feature engineering approach in the context of affect and behavior modeling. We built detectors of student affective states and behaviors as middle school students learned science in an open-ended learning environment called Betty's Brain, using both approaches. Overall, we observed a tradeoff where the feature engineering models were better when considering a single optimized threshold (for intervention), whereas the deep learning models were better when taking model confidence fully into account (for discovery with models analyses).

This poster presents design implications related to findings from research on the development of Interactive Learning Assessments (ILAs). Standardized assessments using multiple choice questions cannot measure the most critical aspects of learning for the 21st century. ILAs place students in advisory roles, leveraging a place-based metaphor to navigate, learn, then give counsel in situations in which they do not already know how to solve the problem; we assess how students learn to learn.