Despite large literature on Cross-Cultural Competence (3C) there is a gap in understanding learning processes and mechanisms by which people arrive at successful 3C. We present a novel perspective for 3C learning and decision-making in innovative assessment contexts. We use Mindset theory (i.e., believing ability is fixed or changeable) because it is shown to be a powerful motivator for general learning and performance and in cross-cultural contexts. We propose the notion of cultural mindsets – beliefs, affect, and cognition that govern how people adapt, learn, and update cultural information. To understand how cultural mindset affects learning and performance, we apply computational cognitive modeling using Markov decision process (MDP). Using logfile data from an interactive 3C task, we operationalize behavioral differences in actions and decision making based on Mindset theory, developing cognitive models of fixed and malleable cultural mindsets based on mechanisms of initial beliefs, goals, and belief updating. To explore the validity of our theory, we develop computational MDP models, generate simulated data, and examine whether performance patterns fit our expectations. We expected the malleable cultural mindset would be better at learning the cultural norms in the assessment, more persistent in cultural interactions, quit less before accomplishing the task goal, and would be more likely to modify behavior after negative feedback. We find evidence of distinct patterns of cultural learning, decision-making, and performance with more malleable cultural mindsets showing significantly greater cultural learning, persistence, and responsiveness to feedback, and more openness to exploring current cultural norms and behavior. Moreover, our model was supported in that we were able to accurately classify 83% of the simulated records from the generating model. We argue that cultural mindsets are important mechanisms involved in effectively navigating cross-cultural situations and should be considered in a variety of areas of future research including education, business, health, and military institutions.

Assessments built on a theory of learning progressions are promising formative tools to support learning and teaching. The quality and usefulness of those assessments depend, in large part, on the validity of the theory-informed inferences about student learning made from the assessment results. In this study, we introduced an approach to address an important challenge related to examining theorized level links across progressions. We adopted the method to analyze response data for three learning progressions: Equality and Variable, Functions and Linear Functions, and Proportional Reasoning in middle-school mathematics . Multidimensional item response theory models were fit to the data to evaluate the postulated learning levels and level links. Our findings supported the theoretical hypotheses for Functions and Linear Functions, and Proportional Reasoning. Thus, items measuring these progressions can be used to develop formative assessment tasks, and assist instructional practices. Our findings did not support the theory underlying Equality and Variable. Implications for assessment developers and users, and future directions for research were discussed.

The goal of the present work is to build a foundation for understanding cognition and decision-making processes in innovative assessment contexts. Specifically, we will assess students’ Cross-Cultural Competence (3C: see Thomas et al., 2008) through a social simulation game. The present work will use Mindset (i.e., individuals beliefs about whether ability is fixed or changeable, see Dweck, 2006) to ground the project in theory because it has been shown to be a powerful motivator for decision-making and behavior in learning and achievement (Dweck & Leggett; 1988; Dweck, 1999), and in cross-cultural contexts (Dweck, 2012). The novel contribution of this paper is to apply Mindset theory to social situations requiring 3C, thus proposing the notion of cultural mindsets—defined here as the set of beliefs including affect, cognition, and behavior people bring to cross-cultural contexts. In cultural mindset, affect and cognition govern the ease with which people adapt, learn, and update cultural information. Additionally, we argue that cultural mindsets are important mechanisms involved in navigating cross-cultural situations effectively and should be considered more in future research. In order to understand how cultural mindset affects student performance, we will apply a computational cognitive modeling approach using Markov decision process (MDP) models. The MDP approach is appropriate for sequential decision-making in non-deterministic environments—as actions are chosen as part of a plan to achieve goals with the knowledge that some action effects will be probabilistic.

In this article, we report on a three-pronged effort to create a hypothetical learning progression for quantification in science. First, we drew from history and philosophy of science to define the quantification competency and develop hypothetical levels of the learning progression. More specifically, the quantification competency refers to the ability to analyze phenomena through (a) abstracting relevant measurable variables from phenomena and observations, (b) investigating the mathematical relationships among the variables, and (c) conceptualizing scientific ideas that explain the mathematical relationships. The quantification learning progression contains four levels of increasing sophistication: level 1, holistic observation; level 2, attributes; level 3, measurable variables; and level 4, relational complexity. Second, we analyzed the practices in the Next Generation Science Standards for current, largely tacit, assumptions about how quantification develops (or ought to develop) through K-12 education. While several pieces of evidence support the learning progression, we found that quantification was described inconsistently across practices. Third, we used empirical student data from a field test of items in physical and life sciences to illustrate qualitative differences in student thinking that align with levels in the hypothetical learning progression for quantification. By generating a hypothetical learning progression for quantification, we lay the groundwork for future standards development efforts to include this key practice and provide guidance for curriculum developers and instructors in helping students develop robust scientific understanding.

In this paper, a learning progression for geometric transformations is developed based on research that demonstrates the importance of viewing transformations as functions of the plane. The 5 levels of the progression reflect a student's evolving understanding of transformations as functions and their evolving understanding of the domain of these transformation as functions. The learning progression developed here is designed to be in alignment with the Common Core State Standards in Mathematics (CCSSM), in that a student who is placed at a particular level in the learning progression would have mastered the Common Core standards at the corresponding grade level. The description of the learning progression also includes sample tasks at each level that are intended to target that level of the progression.

The interplay between games, assessment, and learning has so far been considered primarily within a formative setting and with particular focus on how design frameworks from these fields are connected and disconnected. The purpose of this chapter is to extend what has been developed and learned about formative game-based assessments (GBAs) into summative assessment practices. The objective is to answer questions about design principles, good or best practices, and application opportunities for summative game-based assessments, including a critical analysis of how they can or cannot improve summative assessment practices more generally. After introducing some foundations, we will discuss motivations for GBA, provide design trade-offs for various use cases, and develop considerations for designing summative GBAs. Careful review shows that this is a very challenging space with some limited, though worthwhile opportunities.

This article provides a validation framework for research on the development and use of science Learning Progressions (LPs). The framework describes how evidence from various sources can be used to establish an interpretive argument and a validity argument at five stages of LP research-development, scoring, generalisation, extrapolation, and use. The interpretation argument contains the interpretation (i.e. the LP and conclusions about students' proficiency generated based on the LP) and the use of the LP. The validity argument specifies how the evidence from various sources supports the interpretation and the use of the LP. Examples from our prior and current research are used to illustrate the validation activities and analyses that can be conducted at each of the five stages. When conducting an LP study, researchers may use one or more validation activities or analyses that are theoretically necessary and practically applicable in their specific research contexts.

This paper explores the relation between formative assessment principles and their analogues in video games that game designers have been developing over the past 35 years. We identify important parallels between the two that should enable effective and efficient use of well-designed video games in the classroom as part of an overall learning experience organized and facilitated by teachers. We describe the parallels and then show how game-design elements are used in the service of formative assessment principles in Mars Generation One (MGO): Argubot Academy™, a video game developed by GlassLab that focuses on formative assessment of middle school students’ argumentation skills. MGO was designed and developed together with a broader curricular unit on argumentation within which the game was situated. We discuss how design elements in the game satisfy each core formative assessment principle and how the game is connected to the instructional unit, demonstrating this marriage of game design and formative assessment. Prior work has reported on preliminary evidence on efficacy of the game when used as part of this middle school language arts unit.

Formative feedback is well known as a key factor in influencing learning. Modern interactive learning environments provide a broad range of ways to provide feedback to students as well as new tools to understand feedback and its relation to various learning outcomes. This issue focuses on the role of formative feedback through a lens of how technologies both support student learning and enhance our understanding of the mechanisms of feedback. The papers in the issue span a variety of feedback strategies, instructional domains, AI techniques, and educational use cases in order to improve and understand formative feedback in interactive learning environments. The issue encompasses three primary themes critical to understanding formative feedback: 1) the role of human information processing and individual learner characteristics for feedback efficiency, 2) how to deliver meaningful feedback to learners in domains of study where student work is difficult to assess, and 3) how human feedback sources (e.g., peer students) can be supported by user interfaces and technology-generated feedback.

Extracting information efficiently from game/simulation-based assessment (G/SBA) logs requires two things: a well-structured log file and a set of analysis methods. In this report, we propose a generic data model specified as an extensible markup language (XML) schema for the log files of G/SBAs. We also propose a set of analysis methods for identifying useful information from the log files and implement the methods in a package in the Python programming language, glassPy. We demonstrate the data model and glassPy with logs from a game-based assessment, SimCityEDU.

The Cognitive Science Research Group at Educational Testing Service (ETS) Irvin R. Katz Educational Testing Service Malcolm Bauer Educational Testing Service Gabrielle Cayton-Hodges Educational Testing Service Gary Feng Educational Testing Service G. Tanner Jackson Educational Testing Service Madeleine Keehner Educational Testing Service Juan Diego Zapata-Rivera Educational Testing Service Abstract: The Cognitive Science Research group at ETS conducts research and development at the forefront of educational assessment, using cognitive theory in the design of assessments, building cognitive models to guide interpretation of test-takers’ performance, and researching cognitive issues in the context of assessment. Moving beyond traditional (e.g., multiple-choice) tests, the group investigates reliable and valid assessment (both summative and formative) using innovative, highly interactive digital environments such as online games, virtual labs or other simulations, and human-agent conversation-based interactions. Researchers also investigate how to draw appropriate inferences about test-takers’ knowledge and skills from complex data sources such as eye-movements, interaction logs, and other sequential information. I will provide an overview of the group’s research, including the use of cognitive models to interpret test-takers’ actions within interactive assessment tasks and empirical studies on how people coordinate between internal and external representations during assessments in domains such as writing and scientific inquiry.

Almond, Kim, Velasquez, and Shute explore a key part of evidence-centered design (ECD)— 6 different but nonexclusive roles that task model variables can have and their function within an overall assessment design. The authors provide examples of these roles for more-conventional tasks (math word problems) and for a game-based formative assessment, Newton’s Playground. The article focuses on structure—the roles of these variables and the relation of these variables to the rest of the assessment design. This commentary addresses aspects of ECD that are complementary to the article, specifically aspects of process (How does one create a design involving task model variables? What is the dynamic relation between the student-model, evidence-model, and task-model variables as they are developed and revised?). While the structures described in Almond et al. provide the “what” of evidence-centered assessment design, the intent of this commentary is to discuss more about the design process itself—that is, the “how” that leads to these useful structures. Difficult assessment-design problems, like many difficult engineering-design problems, typically need to optimize multiple criteria (e.g., measurement characteristics, user experience, aspects of validity) under many constraints (e.g., development time, testing time, delivery infrastructure, cost). From this perspective, the roles for task model variables that Almond et al. identify and illustrate are elements of the design space that can be manipulated to help optimize criteria or satisfy constraints and solve the assessment-design problem at hand. The process of designing an assessment, especially one that involves innovation in measurement, typically involves addressing these multiple criteria and constraints simultaneously, yet also iteratively at successive levels of detail. Mislevy, Steinberg, and Almond (1999) built this into the ECD process across the 3 stages of ECD: domain analysis, domain modeling, and the conceptual assessment framework. Each stage addresses successively more-specific questions about the competencies, evidence, and tasks and about their relation to each other.

This chapter describes the characteristics of games and how they can be applied to the design of. innovative assessment tasks for formative and summative purposes. Examples of current. educational games and game-like assessment tasks in mathematics, science, and English language learning arc used to illustrate some of these concepts. We argue that the inclusion of some aspects from gaming technology may have a positive effect in the development of innovative assessment systems (e.g., by supporting the development of highly engaging assessment tasks). However, integrating game elements as part of assessment tasks is a complex process that needs to take into account not only the engaging or motivational aspects of the activity but also the quality criteria that are needed according to the type of assessment that is being developed.