Rubrics support students in learning complex generic (21st century) skills, as they provide textual descriptions of skills’ mastery levels with performance indicators for all constituent subskills. If students know their current and strived-for mastery level, they can better determine subsequent learning activities towards skills mastery. However, text-based rubrics have a limited capacity to support the formation of mental models of a complex skill. Video-enhanced rubrics (VER) with video modeling examples have the potential to improve and enrich mental model formation, feedback quality, and thus improve students’ performance. In the Viewbrics-project we therefore developed, through design-based research, a methodology for the formative assessment of complex skills with Video-Enhanced Rubrics (VER), precipitated in an online tool. This paper describes the features of the first prototype of this online tool and the results of a stakeholder evaluation of its perceived usefulness and usability, by means of a questionnaire and card-sorting exercise, with 7 teachers and 21 students of two secondary schools. The evaluation of this first prototype showed that both teachers and students evaluated the online tool and formative assessment methodology as handy, usable, helpful and feasible for learning complex skills, although some recommendations were made to further improve the design of the tool.
Automated generation of assessment items can provide large item pools for formative assessments with little effort. However, if the generation process produces self-contained items, these need to be updated or re-generated each time the data source used for generation changes. This paper describes and discusses an alternative approach that dynamically retrieves item content from Wikidata using SPARQL queries. The paper compares four different examples and discusses both benefits and limitations of this approach. Results show that the approach is usable for a broad range of different items for formative assessment scenarios and that limitations are manageable with acceptable effort.
To cope with changing demands from society, higher education institutes are developing adaptive curricula in which a suitable integration of workplace learning is an important factor. Automated feedback can be used as part of formative assessment strategies to enhance student learning in the workplace. However due to the complex and diverse nature of workplace learning processes, it is difficult to align automated feedback to the needs of the individual student. The main research question we aim to answer in this design-based study is: ‘How can we support higher education students’ reflective learning in the workplace by providing automated feedback while learning in the workplace?’. Iterative development yielded (1) a framework for automated feedback in workplace learning, (2) design principles and guidelines and (3) an application prototype implemented according to this framework and design knowledge. In the near future, we plan to evaluate and improve these tentative products in pilot studies.
Conversational interfaces (also called chatbots) have recently disrupted the Internet and opened up endless opportunities for assessment and learning. Formative feedback that provides learners with practical instructions for improvement is one of the challenging tasks in self-assessment settings and self-directed learning. This becomes even more challenging if a user’s personal information such as learning history and previous achievements cannot be exploited for data protection reasons or are simply not available. This study seeks to explore the opportunities of providing formative feedback in chatbot-based self-assessment. Two main challenges were faced: the limitations of the messenger as an interface that restricts visual representation of the quiz questions, and zero information about the user to generate adaptive feedback. Two types of feedback were investigated regarding their formative effect: immediate feedback, which was given after answering a question, and cumulative feedback detailing strengths and weaknesses of the user in each of the topics covered along with the directives for improvement. A chatbot called SQL Quizbot was deployed on Facebook Messenger for the purposes of this study (Try out the prototype at https://www.messenger.com/t/2076690849324267). A survey conducted to disclose users’ perception of the feedback reveals that more than 80% of the users find immediate feedback helpful. Overall this study shows that chatbots have a great potential as an aiding tool for e-learning systems to include an interactive component into feedback in order to increase user motivation and retention.
Modern technologies increasingly make use of personal data to provide better services. Technologies using biometric data for identity and authorship verification in the context of e-assessment are a case in point. Previous studies in e-health described a privacy paradox in relation to consent to personal data use: even when people consider protection of their personal data important, they consent fairly readily to personal data use. However, the new European Data Protection Regulation (GDPR) assumes that people give free and informed consent. In the context of e-assessment, this study investigates students’ attitudes towards personal data sharing for identity and authorship verification purposes with the aim of optimising informed consent practice. Students with special educational needs or disabilities (SEND) were included as a specific target group because they may feel more dependent on e-assessment. The findings suggest that a privacy paradox exists in the context of e-assessment as well. Furthermore, the results indicate that students are more reluctant to share video recordings of their face than other personal data. Finally, our results confirm the effect found in previous studies on e-health: those feeling a stronger need for technologies, in this case SEND students, are more inclined to consent to personal data use. Implications for informed consent practice are discussed.
Item banks in higher education can be regarded as important assets to increasing the quality of education and assessment. An item bank allows for the flexible administration of computer-based achievement tests for summative purposes, as well as quizzes for formative purposes. Developing item banks, however, can require quite an investment. A well-worked-out business case can help with convincing stakeholders to start an item bank development project. An important part of such a business case should be the increase in item quality and the estimated reduction in costs, particularly for the collaborative development of an item bank. However, a theoretical underpinning of a business case, incorporating considerations based on classical test theory is lacking in the literature. Therefore, a model is described to make estimations of reductions in misclassifications and per-unit costs. Examples are presented of the likelihood of reducing misclassifications and cost per unit based on findings in the literature. Implications for research and practice are discussed.
Automatic Assessment Systems empowered by mathematical engines allow the development of online assignments for Mathematics, which goes beyond multiple-choice modality. Automatically assessed assignments, used with formative purposes, can support teaching and learning from several perspectives, such as conceptual and procedural understanding, metacognition, enactment of adaptive strategies, and teachers’ management of the class. This paper reports on an experimentation where automatic assessment has been used in a blended modality according to a model of formative assessment and interactive feedback to enhance learning. The experiment involved a total number of 546 students of 8th grade in the town of Turin (Italy). The use of the automatic assessment is shown and exemplified. Data from learning tests, questionnaire and platform usage are analyzed and used to show the effectiveness of the interactive materials for enhancing mathematical understanding and self-assessment skills. Moreover, a profile of the students who did not use the online opportunities, defined as “reluctant users”, is drawn and discussed.
This paper proposes gesture performance as one main channel for assessing collaboration skills, while multiple users solve a problem collaboratively on a tangible user interface. Collaborative problem solving incorporates two dimensions, complex problem solving and collaboration. Thus, the technology-based assessment of collaborative problem solving includes assessing both problem solving and collaboration skills. Particularly, for assessing collaboration skills, we consider gesture performance as an important indicator. We differentiate between physical 3D mid-air gestures and manipulative gestures; for the latter, we developed a gesture recognition application using Kinect. The method we follow for object and gesture recognition is to merge the logging files from our tangible interface software framework (object recognition) with the Kinect log files (gesture recognition) in one file. The application can analyze the number of object manipulations with respect to timing axis, subject/participant, and handedness.
This exploratory study examines the experience and views of students about 3D Virtual Reality Glasses (3DVRG) and e-authentication systems. The authors developed the "Virtual Reality Classroom" App, which is an Open Educational Resource based on 360 photos of the renowned "Bletchley Park". Participants were 2 groups of students from the UK and Brazil who explored in pairs this App using a 3DVRG in the classroom and also completed a formative assessment activity using the TeSLA face recognition system. Our research question focuses on whether the students' interactions through the 3DVRG enhance learning and assessment of inquiry skills for Responsible Research and Innovation (RRI). Findings suggested that the combination of authentic scenario, interactive tasks and assessment-in-context helped students acquire new information and connect with their existing knowledge and practice the inquiry skills together. These interactions enhanced their immersion, particularly for those who found the activities fun as they did not experience motion sickness. Three types of interactions were identified between students with: the virtual space (1), their peer (2) and the topic (3). These three interactions propitiated, respectively, students virtual, social and cognitive presence, which supported their experiential learning.
Digital exams are rather uncommon in engineering education because general e-assessment platforms lack the ability to use advanced item types that mimic general engineering problem-solving processes and award partial scores. However, it is possible to develop such advanced items with Maple T.A.. We describe how such items are structured in scenarios and developed for a second year bachelor’s-level material science course that ran three times at the Delft University of Technology. We evaluate how these items function in practice, are scored and perform from an educational measurement perspective. The paper discusses the results of the study and future directions for development of digital exams in engineering courses.
This paper reports on practical experiences with the two e-assessment tools AlephQ and JACK, explains their key features and sketches usage scenarios from two different universities. Using a lecture in accountancy as a concrete example, the paper then presents a successful concept for improving a lecture by introducing both e-assessment systems. Conclusions are drawn on how to improve a lecture by selecting and combining the most suitable features from different tools.
Students' confidence about their knowledge may yield high or low discrepancy in contrast to actual performance. Therefore, investigating students' behavior towards corrective feedback (received after answering a question) becomes of particular interest. We conducted three experimental sessions with 94 undergraduate students using a computer-based assessment system wherein students specified confidence level (as high or low) with each submitted response. This research study exploits their logged data to provide analyses of: (1) students' behaviors towards corrective feedback in relation to their confidence (about his/her answers), and, (2) impact of seeking corrective feedback on student's subsequent attempt. In conformance with previous studies, we determine that students tend to overestimate their abilities. Data analysis also shows a significant difference infv students' feedback seeking behavior with respect to distinct confidence-outcome categories. Interestingly, feedback seeking was predicted by (student) response's outcome irrespective of its related confidence level, whereas, feedback reading time shows dependency on the confidence level. Our most important finding is that feedback seeking behavior shows a positive impact on students' confidence-outcome category in the next attempt. Different possibilities for utilizing these results for future work and supporting adaptation based on students' needs are discussed in the conclusions.
This design research was focused on developing standards covering the entire process of examination to limit the chances of security risks (e.g., the prevention of exam fraud as much as possible, and detection by means of data forensics), together these standards form the Educational Data Forensics Protocol. Two research questions guided this study. The first question was, which standards regarding preventing and detecting fraud in the process of examination need to be included into the EDF protocol? In addition, practitioners must be able to act on indications of exam fraud based on these standards. Therefore, a second research question was formulated, namely which conditions must be considered during development of the EDF protocol to support practitioners in detecting possible gaps in the security of their examination process? The EDF protocol was developed and validated in five consecutive steps. This study analyses on the theoretical base of developing the EDF protocol (Step 1) and the considerations for developing a prototype (Step 2). The prototype was being validated (e.g., establishing correctness of the content) through seven semi-structured interviews with content experts in the field of either test security or data forensics (Step 3). Statements from these interviews were used to adjust the prototype into a final version of the EDF protocol (Step 4). Finally, to determine the practical value, the final version of the EDF protocol was used to flag gaps in the security of the exam process and determine possible security risks for one of eX:plain’ s exam programs (Step 5).
This paper presents a retrospective analysis of learning in a MOOC as reconstructed from the conversations that learners conducted in MOOC group forums while performing the course tasks. A mixed method approach was applied to analyze the quantity and the quality of these conversations. Two activity patterns were distinguished – in groups with higher activity levels, there were more individual contributions (posts) on more course themes and these contributions were broader spread throughout the course. In high activity groups there was also more interaction between participants, i.e., more questions, answers, explanations and elaborations. The presented study demonstrates how modeling interactions in group forums helps to elicit individual and emerging group knowledge construction and thus supports defining MOOC learning, informs MOOC design and provides insights on how assessing MOOC learning can be automated.
We present some preliminary findings of the Erasmus+ KA2 Strategic Partnership project "Online Proctoring for Remote Examination" (OP4RE). OP4RE aims to develop, implement and disseminate up to par practices for remote examination procedures. More specifically, OP4RE strives to develop guidelines and minimum standards for the secure, legal, fair and trustworthy administration of exams in a remote location away from physical exam rooms in a European context. We present findings and issues regarding security, cheating prevention and deterrence, privacy and data protections as well as practical implementation.
This paper reports on an essay-writing study using a technical system that has been developed to generate automated feedback on academic essays. The system operates through the combination of a linguistic analysis engine, which processes the text in the essay, and a web application that uses the output of the linguistic analysis engine to generate the feedback. In this paper we focus on one particular visual representation produced by the system, namely “rainbow diagrams”. Using the concept of a reverse rainbow, diagrams are produced which visually represent how concepts are interlinked between the essay introduction (violet nodes) and conclusion (red nodes), and how concepts are linked and developed across the whole essay – thus a measure of how cohesive the essay is as a whole. Using a bank of rainbow diagrams produced from real essays, we rated the diagrams as belonging to high-, medium- or low-scoring essays according to their structure, and compared this rating to the actual marks awarded for the essays. On the basis of this we can conclude that a significant relationship exists between an essay’s rainbow diagram structure and the mark awarded. This finding has vast implications, as it is relatively easy to show users what the diagram for a “good” essay looks like. Users can then compare this to their own work before submission so that they can make necessary changes and so improve their essay’s structure, without concerns over plagiarism. Thus the system is a valuable tool that can be utilised across academic disciplines.
This paper presents FLEX, a framework for electronic assessment on students’ devices. Basic requirements to such a framework and potential issues related with these requirements are discussed, as well as their state of the art. Afterwards, the client-server architecture of FLEX is presented, which is designed to meet all requirements previously identified. The FLEX client and the FLEX server are discussed in detail with focus on utilized technologies and programming languages. The results of first trials with the existing prototype are discussed in relation to the identified basic requirements. Thereafter, assessment of programming courses is discussed as use case of FLEX, which makes use of the extensibility of client and server. The paper closes with a summary and an outlook.
The aim of this study is to investigate whether the LiteMap application tool helps teachers annotate students’ socio-scientific discussion and assess their evidence-based dialogue using a rubric system of inquiry skills for Responsible Research and Innovation (RRI). This study focuses on a set of materials and activities of the European ENGAGE project used by Brazilian students from a city affected by the Zika virus to discuss whether the mosquito Aedes Aegypti should be exterminated or not. The Zika virus project was developed by 24 teachers and 478 students from a public professional school in Irecê including also 5 collaborators and 2 researchers. This qualitative study analyses the dialogue of 35 students (21 girls and 14 boys) randomly selected who participated of 1-h debate to discuss their informed views and evidence-based opinions. Findings of this study reveal that the rubric system facilitates the annotation and mapping of questions, claims, arguments and evidence. LiteMap was useful to represent students’ evidence based dialogue and provide feedback. The visualization of evidence-based dialogue maps can be used by teachers and students during formative assessment of inquiry skills for RRI. However, the process must be planned and it requires time.
For online and blended education institutions, there is a severe handicap when they need to justify how the authentication and authorship of their students are guaranteed during the whole instructional process. Different approaches have been proposed in the past but most of them only depend on specific technological solutions. These solutions in order to be successfully accepted in educational settings have to be transparently integrated with the educational process according to pedagogical criteria. This paper analyses the results of the first pilot based on the TeSLA trustworthy system for a blended and a fully online institutions focused on engineering academic programs.
It is very difficult and time consuming to assess texts. Even after great effort there is a small chance independent raters would agree on their mutual ratings which undermines the reliability of the rating. Several assessment methods and their merits are described in literature, among them the use of rubrics and the use of comparative judgement (CJ). In this study we investigate which of the two methods is more efficient in obtaining reliable outcomes when used for assessing texts. The same 12 texts are assessed in both a rubric and CJ condition by the same 6 raters. Results show an inter-rater reliability of .30 for the rubric condition and an inter-rater reliability of .84 in the CJ condition after the same amount of time invested in the respective methods. Therefore we conclude that CJ is far more efficient in obtaining high reliabilities when used to asses texts. Also suggestions for further research are made.
