Assessing group work formatively in higher education poses a significant challenge. The complexity of evaluating individual contributions is compounded by the lack of efficient and effective methods for tracking, analysing and assessing individual engagement and contributions, which can impede timely feedback and the development of group work skills. This paper contributes to the growing body of research on collaboration analytics, which focuses on learning analytics (LA) in collaborative settings, and formative assessment while providing practical guidance for educators seeking to enhance formative assessment practices. The potential enhancement lies at the intersection between analytics technology and assessment design. In this paper, we present a conceptual framework that can harness multimodal data collection as well as LA to formatively assess and provide feedback on individual engagement and contributions to group work across physical and digital spaces. Drawing on research that considers design for learning, conjecture mapping, assessment design, multimodal LA and feedback, we outline a structured approach to developing formative assessment of group work skills in collaborative projects and higher education contexts. Implications for practice or policy: center dot Practical guidance is provided for educators to enhance formative assessment practices with LA. center dot The integrated framework and process offers a guide for formative assessment in group work scenarios, providing tools for monitoring student progress, informing pedagogical decision-making and enhancing learning experiences. center dot The integration of the activity-centred analysis and design framework and LA allows educators to harness the power of both data-driven decision-making and learner- centred pedagogical approaches to better support learner development in group work settings.

This paper documents and shares findings from a Systematic Literature Review (SLR) that was conducted to explore what published evidence exists of how Indigenous perspectives have been incorporated into the Australian tertiary engineering curriculum. The importance of embedding these perspectives into engineering education is known as it provides holistic and authentic learning experiences, reflects real world and industry practices, and better equips graduates. Engineers must competently demonstrate understanding of, and ethical conduct for, social and community duties. In order to provide graduates with the knowledge and awareness of how engineering practices affect human, societal, and environmental demands, it is becoming increasingly important to emphasise the human elements of engineering. The SLR resulted in 27 included results which highlighted six reoccurring themes, vital for the shared success when embedding Indigenous perspectives. This included community engagement, capacity building for engineering educators, resource repository, storytelling, use of frameworks, and projects.

Student Evaluation of Teaching surveys (SETs) are used at universities to inform teaching practice and subject design. However, there is increasing concern about the impact of allegations, abuse, and discrimination in survey open text components. Here we discuss the implementation of an automated screening mechanism using a combination of dictionary and machine learning approaches. We present both a process diagram detailing how the screening is performed, as well as a form of categorisation for comments that are unacceptable or indicate a potential risk of harm. Examples of real comments in each of these categories are presented to demonstrate the depth of the challenge and potential cause for concern. Ultimately, we argue that student and educator wellbeing are inextricably connected and exposing staff to abusive and discriminatory comments causes harm. Furthermore, SETs are an important channel for students to raise concerns about their own wellbeing and potentially unsafe experiences in the learning environment.

Student evaluation of teaching (SET) surveys are the most widely used tool for collecting higher education student feedback to inform academic quality improvement, promotion and recruitment processes. Malicious and abusive student comments in SET surveys have the potential to harm the wellbeing and career prospects of academics. Despite much literature highlighting abusive feedback in SET surveys, little research attention has been given to methods for screening student comments to identify and remove those that may cause harm to academics. This project applied innovative machine learning techniques, along with a dictionary of keywords to screen more than 100,000 student comments made via a university SET during 2021. The study concluded that these methods, when used in conjunction with a final stage of human checking, are an effective and practicable means of screening student comments. Higher education institutions have an obligation to balance the rights of students to provide feedback on their learning experience with a duty to protect academics from harm by pre-screening student comments before releasing SET results to academics.

Teaching evaluation is deeply entrenched in institutional quality assurance and is a feature of a range of policies including academic recruitment, promotion, and performance management. Universities must ensure that evaluation practices meet regulatory requirements, while balancing student voice and wellbeing. There is extensive literature examining the validity, reliability, and bias of student surveys, but limited focus on institutional evaluation strategies and what constitutes best practice. This study analyses and reports on the teaching evaluation strategies and practices of Australian and New Zealand universities. All 29 participating institutions use standardised and centrally deployed surveys to evaluate teaching and participated in external student experience surveys to benchmark nationally. Comparisons are provided between the strategic intent of evaluation; survey practices, technology, analysis, and reporting; and the use of other nonsurvey methods. The study informs higher education institutions' evaluation strategy, policies, and practice and provides an agenda for reframing approaches to evaluation of teaching.

In this study, we applied natural language processing (NLP) techniques, within an educational environment, to evaluate their usefulness for automated assessment of students’ conceptual understanding from their short answer responses. Assessing understanding provides insight into and feedback on students’ conceptual understanding, which is often overlooked in automated grading. Students and educators benefit from automated formative assessment, especially in online education and large cohorts, by providing insights into conceptual understanding as and when required. We selected the ELECTRA-small, RoBERTa-base, XLNet-base and ALBERT-base-v2 NLP machine learning models to determine the free-text validity of students’ justification and the level of confidence in their responses. These two pieces of information provide key insights into students’ conceptual understanding and the nature of their understanding. We developed a free-text validity ensemble using high performance NLP models to assess the validity of students’ justification with accuracies ranging from 91.46% to 98.66%. In addition, we proposed a general, non-question-specific confidence-in-response model that can categorise a response as high or low confidence with accuracies ranging from 93.07% to 99.46%. With the strong performance of these models being applicable to small data sets, there is a great opportunity for educators to implement these techniques within their own classes. Implications for practice or policy: Students’ conceptual understanding can be accurately and automatically extracted from their short answer responses using NLP to assess the level and nature of their understanding. Educators and students can receive feedback on conceptual understanding as and when required through the automated assessment of conceptual understanding, without the overhead of traditional formative assessment. Educators can implement accurate automated assessment of conceptual understanding models with fewer than 100 student responses for their short response questions.

This study examines the utility of a new method of analysing and reporting qualitative student survey comments. Visualisations of the sentiment of and key themes from qualitative student survey comments were developed for 34 subjects from two undergraduate health sciences courses undergoing transformation. The course and subject visualisation reports were provided to academics who were members of the two course transformation teams. Utilising focus groups and semi-structured interview methodology, we examined academic perspectives on the potential usefulness of this new method of analysing and reporting qualitative comments from student evaluations of teaching (SET). Results indicate that visualisations are considered useful for focussing educators' attention on themes in qualitative comments rather than on individual negative comments. However, educators did not support use of the visualisations at this stage, citing concerns about the accuracy of sentiment analysis and the validity of SET surveys. There were many opportunities identified through the focus groups for future improvements to a visualisation technique for reporting student survey results.

Contribution: An automated methodology that provides visualizations of students’ free text comments from course satisfaction surveys. Focusing on sentiment, these visualizations reveal learning and teaching aspects of the course that either may require improvement or are performing well. They provide educators with a simple, systematic way to monitor their courses and make pedagogically sound decisions on teaching strategies. Background: Student course satisfaction surveys often solicit free text comments. This feedback can provide invaluable insights for educators, but because these comments often contain a large amount of data, they cannot easily be acted upon. Existing visualization methods are not suitable for this application, and needed additional capabilities. Research Questions: How can large quantities of student satisfaction data be summarized and visualized? How can these visualizations be used to learn meaningful information about courses? What are the recurring themes across semesters? Methodology: Several methods based on machine learning and text analysis techniques were used to visualize student satisfaction comments. The latent Dirichlet allocation (LDA) statistical method was used to identify aspects of student opinion of a course. The sentiment of the student comments were also identified. This information was then presented visually for educators in a case study that gives examples of these visualizations. Findings: The visualization methods explored provide educators with an overview of aspects and their associated sentiment. The summary visualizations allow easy comparison to be made between courses, or between teaching periods in the same course.

ABSTRACTUsing text analysis, computers can find patterns to determine and extract useful information from a set of text. Exploiting the capabilities of text analysis software can efficiently provide educators with the ability to analyse students’ answers and make better judgement on their performance. Such ability would otherwise be impossible to practically gain, especially for large classes. Because of the availability of a wide range of text analysis software, with varied features, an important question to ask is which software to use, that best suits a certain application, and users’ educational technology skills.This paper evaluates seven selected software packages which can be used for textual analysis. The graphical abstract, Figure 1, shows the outputs of one of the software packages (Leximancer), using the text from this paper as input. The data set and results presented use responses from a university-level test in an electrical engineering topic, which are included in this paper. The test was d...

Being able to provide fast and useful feedback to students is an essential part of the learning process. Multiple-choice questionnaires are tool often used, due to the ease of marking, and therefore rapid feedback to students. Since these however lack detail and confidence when checking understanding, an added textual component can help to verify students' understanding. In this paper we discuss a methodology for evaluating and categorizing students multiple-choice and text justification. This is applied to three examples of students comments and shows the usefulness of the added textual field.

Various text analysis techniques exist, which attempt to uncover unstructured information from text. In this work, we explore using statistical dependence measures for textual classification, representing text as word vectors. Student satisfaction scores on a 3-point scale and their free text comments written about university subjects are used as the dataset. We have compared two textual representations: a frequency word representation and term frequency relationship to word vectors, and found that word vectors provide a greater accuracy. However, these word vectors have a large number of features which aggravates the burden of computational complexity. Thus, we explored using a non-linear dependency measure for feature selection by maximizing the dependence between the text reviews and corresponding scores. Our quantitative and qualitative analysis on a student satisfaction dataset shows that our approach achieves comparable accuracy to the full feature vector, while being an order of magnitude faster in testing. These text analysis and feature reduction techniques can be used for other textual data applications such as sentiment analysis.