Gaming the system, a behavior in which learners exploit a system's properties to make progress while avoiding learning, has frequently been shown to be associated with lower learning. However, when we applied a previously validated gaming detector across conditions in experiments with an algebra tutor, the detected gaming was not associated with reduced learning, challenging its validity in our study context. Our exploratory data analysis suggested that varying contextual factors across and within conditions contributed to this lack of association. We present a new approach, latent variable-based gaming detection (LV-GD), that controls for contextual factors and more robustly estimates student-level latent gaming tendencies. In LV-GD, a student is estimated as having a high gaming tendency if the student is detected to game more than the expected level of the population given the context. LV-GD applies a statistical model on top of an existing action-level gaming detector developed based on a typical human labeling process, without additional labeling effort. Across three datasets, we find that LV-GD consistently outperformed the original detector in validity measured by association between gaming and learning as well as reliability. LV-GD also afforded high practical utility: it more accurately revealed intervention effects on gaming, revealed a correlation between gaming and perceived competence in math and helped understand productive detected gaming behaviors. Our approach is not only useful for others wanting a cost-effective way to adapt a gaming detector to their context but is also generally applicable in creating robust behavioral measures.

Gaming the system, a behavior in which learners exploit a system’s properties to make progress while avoiding learning, has frequently been shown to be associated with lower learning. However, when we applied a previously validated gaming detector across conditions in experiments with an algebra tutor, the detected gaming was not associated with learning, challenging its construct validity. Our iterative exploratory data analysis suggested that some contextual factors that varied across and within conditions might contribute to this lack of association. We present a latent variable model, item response theory-based gaming detection (IRT-GD), that accounts for contextual factors and estimates latent gaming tendencies as the degree of deviation from normative behaviors across contexts. Item response theory models, widely used in knowledge assessment, account for item difficulty in estimating latent student abilities: students are estimated as having higher ability when they can get harder items correct than when they only get easier items correct. Similarly, IRT-GD accounts for contextual factors in estimating latent gaming tendencies: students are estimated as having a higher gaming tendency when they game in less commonly gamed contexts than when they only game in more commonly gamed contexts. IRT-GD outperformed the original detector on three datasets in terms of the association with learning. IRT-GD also more accurately revealed intervention effects on gaming and revealed a correlation between gaming and perceived competence in math. Our approach is not only useful for others wanting to apply a gaming assessment in their context but is also generally applicable in creating robust behavioral measures.

The current study introduces a model for measuring student diligence using online behaviors during intelligent tutoring system use. This model is validated using a full academic year dataset to test its predictive validity against long-term academic outcomes including end-of-year grades and total work completed by the end of the year. The model is additionally validated for robustness to time-sample length as well as data sampling frequency. While the model is shown to be predictive and robust to time-sample length, the results are inconclusive for robustness in data sampling frequency. Implications for research on interventions, and understanding the influence of self-control, motivation, metacognition, and cognition are discussed.

Online crowds are a promising source of new innovations. However, crowd innovation quality does not always match its quantity. In this paper, we explore how to improve crowd innovation with real-time expert guidance. One approach would for experts to provide personalized feed-back, but this scales poorly, and may lead to premature convergence during creative work. Drawing on strategies for facilitating face-to-face brainstorms, we introduce a crowd ideation system where experts monitor incoming ideas through a dashboard and offer high-level “inspirations” to guide ideation. A series of controlled experiments show that experienced facilitators increased the quantity and creativity of workers' ideas compared to unfacilitated workers, while Novice facilitators reduced workers' creativity. Analyses of inspiration strategies suggest these opposing results stem from differential use of successful inspiration strategies (e.g., provoking mental simulations). The results show that expert facilitation can significantly improve crowd innovation, but inexperienced facilitators may need scaffolding to be successful.

Large-scale idea generation platforms often expose ideators to previous ideas. However, research suggests people generate better ideas if they see abstracted solution paths (e.g., descriptions of solution approaches generated through human sensemaking) rather than being inundated with all prior ideas. Automated and semi-automated methods can also offer interpretations of earlier ideas. To benefit from sensemaking in practice with limited resources, ideation platform developers need to weigh the cost-quality tradeoffs of different methods for surfacing solution paths. To explore this, we conducted an online study where 245 participants generated ideas for two problems in one of five conditions: 1) no stimuli, 2) exposure to all prior ideas, or solution paths extracted from prior ideas using 3) a fully automated workflow, 4) a hybrid human-machine approach, and 5) a fully manual approach. Contrary to expectations, human-generated paths did not improve ideation (as meas-ured by fluency and breadth of ideation) over simply showing all ideas. Machine-generated paths sometimes significantly improved fluency and breadth of ideation over no ideas (although at some cost to idea quality). These findings suggest that automated sensemaking can improve idea generation, but we need more research to understand the value of human sensemaking for crowd ideation.

Online crowds are a promising source of new innovations. However, crowd innovation quality does not always match its quantity. One way to improve quality is to enable experts to provide personalized feedback. However, this scales poorly, and may lead to premature convergence during creative work. To deal with these issues, we present IdeaGens, a crowd ideation system that adapts expert facilitation, a successful strategy for improving collaborative creativity in face-to-face brainstorms, to crowd brainstorming. In IdeaGens, experts monitor incoming ideas from the crowd through a dashboard, and offer high-level "inspirations" to guide ideation towards interesting solution themes. In a randomized controlled experiment, crowd workers who receive facilitation through IdeaGens generate significantly more creative ideas that unfacilitated crowd workers.

Many crowd ideation systems seek to gather scores of ideas from people online. However, this often leads to many bad ideas and duplication. A dedicated facilitator who guides exploration of the solution space is a common and effective strategy for optimizing ideation in face-to-face brainstorming, but has not yet been explored in computer-supported crowd ideation. We introduce IdeaGens, a social ideation system for guided crowd brainstorming. IdeaGens divides the crowd into ideation and synthesis tasks, and enables efficient data-driven facilitation of the crowd’s ideation. This work can inform general strategies for shepherding the crowd to produce better results for complex collaborative tasks.