Digital technology can help with providing detailed assessment of mathematical competences. We present the Advise-Me project, in which evidence of the level of mastery is collected from free-form input, without restricting user interaction. We discuss the novel components of our approach, such as the use of a domain reasoner and Bayesian networks for open algebra problems, and an upcoming evaluation study in three countries.

Essen 1 STUDENT MODELS TO GENERATE AUTOMATED FEEDBACK ON INTERMEDIATE STEPS IN SOLVING MATHEMATICAL PROBLEMS Marieke Bor-de Vries, Paul Drijvers, Peter Boon, Sietske Tacoma, Bastiaan Heeren, Johan Jeuring, Niels Steenbergen, Sergey Sosnovsky, Wink van Zon, Jesse Koops, Armin Weinberger, Allison Kolling, Brigitte Grugeon-Allys, Françoise Chenevotot-Quentin, Ben-Manson Toussaint 1 Freudenthal Institute, Utrecht University, Utrecht, The Netherlands, m.bor-devries@uu.nl 2 Faculty of Management, Science & Technology, Open University of the Netherlands, Heerlen 3 Dept. of Inf. and Comp. Sciences, Universiteit Utrecht, Utrecht, The Netherlands 4 Cito, Arnhem, The Netherlands 5 Dept. of Educational Technology, Saarland University, Saarbrücken, Germany 6 Laboratoire de Didactique André Revuz, Université Paris Est Créteil, Paris, Franc

We describe an approach to using ICT for assessing mathematics achievement of pupils using learning environments for mathematics. In particular, we look at fine-grained cognitive assessment of free-form answers to math story problems, which requires determining the steps a pupil takes towards a solution, together with the high-level solution approach used by the pupil. We recognise steps and solution approaches in free-form answers and use this information to update a user model of mathematical competencies. We use the user model to find out for which student competencies we need more evidence of mastery, and determine which next problem to offer to a pupil. We describe the results of our fine-grained cognitive assessment on a large dataset for one problem, and report the results of two pilot studies in different European countries.

Statistics is a challenging subject for many university students. In addition to dedicated methods of didactics of statistics, adaptive educational technologies can also offer a promising approach to target this challenge. Inspectable student models provide students with information about their mastery of the domain, thus triggering reflection and supporting the planning of subsequent study steps. In this article, we investigate the question of whether insights from didactics of statistics can be combined with inspectable student models and examine if the two can reinforce each other. Five inspectable student models were implemented within five didactically grounded online statistics modules, which were offered to 160 Social Sciences students as part of their first-year university statistics course. The student models were evaluated using several methods. Learning curve analysis and predictive validity analysis examined the quality of the student models from the technical point of view, while a questionnaire and a task analysis provided a didactical perspective. The results suggest that students appreciated the overall design, but the learning curve analysis revealed several weaknesses in the implemented domain structure. The task analysis revealed four underlying problems that help to explain these weaknesses. Addressing these problems improved both the predictive validity of the adjusted student models and the quality of the instructional modules themselves. These results provide insight into how inspectable student models and didactics of statistics can augment each other in the design of rich instructional modules for statistics.

Architects need insights on the extent to which quality attributes are satisfied in order to adequately evolve software systems. This is especially true for software products, which are delivered to many customers and undergo multiple releases, thereby offering ample opportunities for re-design. Available techniques to validate quality attributes either rely on workshops with stakeholders or are based on design-time software artifacts. Many quality attributes, however, are better assessed at runtime when the software system is in operation. In this paper, we present an approach that enables the systematic processing and interpretation of software operation data to gain architectural knowledge about quality attributes. In addition to introducing this approach—which we call Architectural Intelligence—, we present through a case study on an e-Learning environment a formal framework based on process mining that enables the development of second-order information systems for analyzing software operation data to provide architectural intelligence.

Due to the complexity of the topic and a lack of individual guidance, introductory statistics courses at university are often challenging. Automated feedback might help to address this issue. In this study, we explore the use of student models to provide feedback. The research question is how student models can be used to generate feedback to university freshman in an online course on statistical sampling. An online activity was designed and delivered to 40 Biology freshmen. Instruments for generating student models were designed and student models were generated. Four students were interviewed about the generated models, and about the differences with their own estimation of their understanding. Results show that it is possible to generate individual feedback from student work in an online learning activity and suggest that discussing differences between own estimations and generated student models can be a fruitful teaching strategy.

Online resources are widely used for educational purposes, such as the training of skills. For algebra education in particular, online resources are expected to contribute to skill mastery in an efficient and effective way. However, studies that underpin these claims through a randomized experiment are scarce. To experimentally investigate the effect of online tasks for algebra, sixteen teachers each taught two grade 8 algebra classes, one randomly assigned traditional teaching and the other using an online algebra environment. In total, 842 students took part in a pretest, two posttests, and a retention test. Results show that the experimental group scored slightly below the control group on these tests. The main factors involved are students’ pretest scores and the schools’ experience with ICT. Possible explanations include a spill-over effect and a more superficial type of learning than expected in the experimental condition. These results do not confirm the hypotheses on the effectiveness of using online resources for algebra.

The concept of function is a central but difficult topic in secondary school mathematics curricula, which encompasses a transition from an operational to a structural view. The question in this paper is how to design and evaluate a technology-rich learning arrangement that may foster this transition. With domain-specific pedagogical knowledge on the learning of function as a starting point, and the notions of emergent modeling and instrumentation as design heuristics, such a learning arrangement was designed for grade 8 students and field tested. The results suggest that these design heuristics provide fruitful guidelines for the design of both a hypothetical learning trajectory and concrete tasks, and can be generalized to other design processes.

The integration of digital technology into secondary mathematics education is not yet a widespread success. As teachers are crucial players in this integration, an important challenge is not only to attract early adopters, but also to support mid-adopting teachers in their professional development on this point. The questions addressed in this Chapter are: which practices such mid-adopting teachers develop when starting to use technology in their mathematics classroom; and how these practices change over time while engaging in a project with colleagues and researchers. To answer these questions, theoretical notions of instrumental orchestration, TPACK and community of practice underpin the case study of two mathematics teachers from a group of twelve, who engaged in a project on technology-rich teaching. The data includes lesson observations, blogs and results from questionnaires. The results show the type of teaching practices the teachers develop and the changes in these practices. Even if these changes are modest and the impact of the community is limited, the teachers clearly became more confident in integrating technology in their teaching.

Digital resources offer opportunities to improve mathematics teaching and learning, but meanwhile may question teachers’ practices. This process of changing teaching practices is challenging for teachers who are not familiar with digital resources. The issue, therefore, is what teaching practices such so-called ‘mid-adopting’ mathematics teachers develop in their teaching with digital resources, and what skills and knowledge they need for this. To address this question, a theoretical framework including notions of instrumental orchestration and the TPACK model for teachers’ technological pedagogical content knowledge underpins the setting-up of a project with twelve mathematics teachers, novice in the field of integrating technology in teaching. Technology-rich teaching resources are provided, as well as support through face-to-face group meetings and virtual communication. Data include lesson observations and questionnaires. The results include a taxonomy of orchestrations, an inventory of skills and knowledge needed, and an overview of the relationships between them. During the project, teachers do change their orchestrations and acquire skills. On a theoretical level, the articulation of the instrumental orchestration model and the TPACK model seems promising.

The concept of function is a central but difficult topic in secondary school mathematics curricula, which encompasses a transition from an operational to a structural view. The question in this paper is how the use of computer tools may foster this transition. With domain-specific pedagogical knowledge on the learning of function as a point of departure and the notions of emergent modeling and instrumentation as design heuristics, a potentially rich technology-intensive learning arrangement for grade 8 students was designed and field-tested. The results suggest that the relationship between tool use and conceptual development benefits from preliminary activities, from tools offering representations that allow for progressively increasing levels of reasoning, and from intertwinement with paper-and-pencil work.

The paper concerns the way teachers use technological tools in their mathematics lessons. The aim is to investigate the explanatory power of the theory of instrumental orchestration through its confrontation with a teaching episode. An instrumental orchestration is defined through a didactical configuration, an exploitation mode and a didactical performance. This model is applied to a teaching episode on the concept of function, using an applet embedded in an electronic learning environment. The results suggest that the instrumental orchestration model is fruitful for analysing teacher behaviour, particularly in combination with additional theoretical perspectives.

The availability of technology in the mathematics classroom challenges the way teachers orchestrate student learning. Using the theory of instrumental orchestration as the main interpretative framework, this study investigates which types of orchestrations teachers develop when using technology and to what extent these are related to teachers’ views on mathematics education and the role of technology therein. Data consisted of videotapes of 38 lessons taught by three teachers, who also provided information on their views through questionnaires and interviews. Qualitative analysis of these data led to the identification of orchestration types and teacher profiles. The orchestration preferences of the three teachers proved to be related to their views. A detailed analysis of one exemplary episode suggests how other theoretical perspectives might complement the theory of instrumental orchestration.

Developing ways to use educational technology is not evident for mathematics teachers. With the theory of instrumental orchestration as a framework, we investigate the types of orchestrations three teachers use in a lesson series for grade 8 on the function concept, employing an applet embedded in a digital learning environment. The results show six types of orchestrations that differ in their ICT specificity. Interview data suggest that teachers' preferences for types of orchestrations can be related to their views on mathematics learning and teaching.