Generative AI is widely available and has raised the expectation that it will impact Education. Models, such as ChatGPT, can quickly produce plausible texts on a wide range of topics, and this capability may be of potential use in course content production. This paper selects several important course content production tasks, describes the prompts used, and assesses the quality of the automatically generated texts by a team of experts. Across all tasks, the Generative AI produced content that could help solve specific tasks by aiding with brainstorming, creating outlines, and adhering to particular writing guidance. In all cases, the content required adjustments and checking by human experts.

The aim of this narrative review is to understand how Artificial Intelligence (AI), specifically generative AI (e.g. ChatGPT) is thought to be used to support teachers in content authoring and curriculum production. Our findings indicate that Generative AI is envisioned as a co-design partner in enhancing educational content such as rubrics, lesson plans, interactive exercises, analogies, reflective questions, and case studies tailored to learning goals and instructional needs. The literature suggests that the key issues with the use of ChatGPT in education can be summarised mainly around its accuracy, and reliability, and its potential to lead to plagiarism. The analysis of empirical studies on the use of generative AI in higher education revealed first insights into themes such as ChatGPT's role in administrative tasks, varied perceptions of AI, generational differences in attitudes, multifunctional roles of ChatGPT, concerns about academic integrity and need for AI literacy, and the call for teaching philosophy reform, around assessment.

The wide availability of generative artificial intelligence (AI) for content production has resulted in a growing interest in the area of education, particularly for course content production purposes. This research has mapped out a set of curriculum production tasks and illustrated how generative AI can support three important tasks: the development of course outlines and content, the drafting of assessment instructions and the mapping of learning outcomes to benchmark statements. We evaluated the outputs of the generative AI with five experts: a course production expert, an academic expert, two Learning Design experts and an AI expert. The results indicate that generative AI enabled the generation of plausible content skeletons for content and first drafts of relevant content to aid the course production team’s brainstorming but also highlighted the importance of reviewing generated content. Our research indicates that generative AI can result in shifts in the delivery of these tasks.

Reflection writing is a common practice in higher education. However, manual analysis of written reflections is time-consuming. This study presents an automated analysis of reflective writing to analyze reflective writing in CS education based on conceptual Reflective Writing Framework (RWF) and application of natural language processing and machine learning algorithm. This paper investigates two groups of features extraction (n-grams and PoS n-grams) and random forest (RF) algorithm that utilize such features to detect the presence or absence of the seven indicators (description of an experience, understandings, feelings, reasoning, perspective, new learning, and future action). The automated analysis of reflective writing is evaluated based on 74 CS student essays (1113 sentences) that are from the final year project reports in CS’s students. Results showed the seven indicators can be reliably distinguished by their features and these indicators can be used in an automated reflective writing analysis for determining the level of students’ reflective writing. Finally, we consider the implications of how the conceptualization of providing individualized learning support to students in order to help them develop reflective skills.

Reflective writing is important in higher education that supports students to improve their critical thinking. However, Despite the widespread of using reflective writing in education, there is a lack of literature relating to the important aspects of reflection in computer science (CS) education. This study approached to (a) address theoretically how reflective writing can be in CS (b) investigate theoretically the conceptual reflective writing framework (RWF) indicators in CS. The literature showed that the RWF’s indicators is theoretically sound that the use of specific indicator can be less or more important based on the goal and the context of writing reflectivity. Finally, we consider the implications of refection aspects writing can be customized assessment to students based on the field.

The need for effective Intelligent Tutoring Systems (ITSs) and automated assessment is increasing. One area of ITSs has become urgent is that of the automated assessment of reflective writing. The reflective writing has been promoted, in higher education, in order to encourage students to think critically about their learning. However, many frameworks have been developed for assessing student’s reflective writing. Up to our knowledge, there is no empirical studies to validate reflective writing frameworks that used in Computer Science (CS) education. This paper presents the validation of reflective Writing Framework (RWF) by CS educators. The expert panelists validated the RWF. Subsequently, we proposed an ITS model for automating reflective writing analysis. The RWF was accepted that it received a level of consensus from the experts who reported obtaining from good to appropriate results using it.

The term ’awareness’ finds it roots in the research on Computer Supported Cooperative Work (CSCW) and often does not reflect the changed modi operandi in today’s connected world. In this paper we argue that the term ’awareness’ needs to be understood in a broader way when used in the context of networked learning. In Learning Networks, awareness is increasingly related to finding appropriate learning objects, peers and experts or the ’right’ learning path. We discuss this different understanding and formulate open questions dealing with awareness in Learning Networks and Personal Learning Environments as well as their connection to reflection and issues of technical feasibility.

The accuracy of a framework for annotating reflective writing can be increased through the evaluation and revision of the annotation scheme to ensure the reliability and validity of the framework. To our knowledge, there is a lack of literature related to the accuracy of any reflective writing framework in Computer Science (CS) education. This paper describes a manual annotation scheme, applied during four pilot studies, to validate the authors' novel Reflective Writing Framework (RWF) for CS education. The results show, through the pilot studies, that the accuracy of Inter-Rater Reliability (IRR) increases from 0.5 to 0.8, which was substantial and close to an almost perfect agreement. This paper contributes to CS education through the reliability and validity of the RWF that can be potentially used for generating an Intelligent Tutoring Systems (ITS) using machine learning algorithms.

Based on a literature review, reflections in written text are rare. The reported proportions of reflection are based on different baselines, making comparisons difficult. In contrast, this research reports on the proportion of occurrences of elements of reflection based on sentence level. This metric allows to compare proportions of elements of reflection. Previous studies are based on courses tailored to foster reflection. The reported proportions represent more the success of a specific instruction than informing about proportions of reflections occurring in student writings in general. This study is based on a large sample of course forum posts of a virtual learning environment. In total 1000 sentences were randomly selected and manually classified according to six elements of reflection. Five raters rated each sentence. Agreement was calculated based on a majority vote. The proportions of elements of reflection are reported and its potential application for course analytics demonstrated. The results indicate that reflections in text are indeed rare, and that there are differences within elements of reflection.

The skill to take part in and to contribute to debates is important for informal and formal learning. Especially when addressing highly complex issues, it can be difficult to support learners participating in effective group discussion, and to stay abreast of all the information collectively generated during the discussion. Technology can help with the engagement and sensemaking of such large debates, for example, it can monitor how healthy a debate is and provide indicators of participation's distribution. A special framework that aims at harnessing the intelligence of - small to very large – groups with the support of structured discourse and argumentation tools is Contested Collective Intelligence (CCI). CCI tools provide a rich source of semantic data that, if appropriately processed, can generate powerful analytics of the online discourse. This study presents a visualisation dashboard with several visual analytics that show important aspects of online debates that have been facilitated by CCI discussion tools. The dashboard was designed to improve sensemaking and participation in online debates and has been evaluated with two studies, a lab experiment and a field study in the context of two Higher Education institutes. The paper reports findings of a usability evaluation of the visualisation dashboard. The descriptive findings suggest that participants with little experience in using analytics visualisations were able to perform well on given tasks. This constitutes a promising result for the application of such visualisation technologies as discourse-centric learning analytics interfaces can help to support learners' engagement and sensemaking of complex online debates.

Especially in lifelong or professional learning, the picture of a continuous learning analytics process emerges. In this process, heterogeneous and changing data source applications provide data relevant to learning, at the same time as questions of learners to data change. This reality challenges designers of analytics tools, as it requires analytics tools to deal with data and analytics tasks that are unknown at application design time. In this paper, we describe a generic visualization tool that addresses these challenges by enabling the visualization of any activity log data. Furthermore, we evaluate how well participants can answer questions about underlying data given such generic versus custom visualizations. Study participants performed better in 5 out of 10 tasks with the generic visualization tool, worse in 1 out of 10 tasks, and without significant difference when compared to the visualizations within the data-source applications in the remaining 4 of 10 tasks. The experiment clearly showcases that overall, generic, standalone visualization tools have the potential to support analytical tasks sufficiently well.

Many disciplines already embed reflective practice in their curriculum as it is important for the professional development of their students. Many frameworks have been developed to capture reflective practice, but there is a lack of knowledge about what constitutes it in computer science education (CSE). This poster introduces a reflective writing framework (RWF) in CSE. This poster makes two contributions: (1) it outlines the construction process of the RWF for CSE, and (2) it discusses the proposed validation of the RWF with a panel of experts.

Reflective writing is an important educational practice to train reflective thinking. Currently, researchers must manually analyze these writings, limiting practice and research because the analysis is time and resource consuming. This study evaluates whether machine learning can be used to automate this manual analysis. The study investigates eight categories that are often used in models to assess reflective writing, and the evaluation is based on 76 student essays (5080 sentences) that are largely from third- and second-year health, business, and engineering students. To test the automated analysis of reflection in writings, machine learning models were built based on a random sample of 80% of the sentences. These models were then tested on the remaining 20% of the sentences. Overall, the standardized evaluation shows that five out of eight categories can be detected automatically with substantial or almost perfect reliability, while the other three categories can be detected with moderate reliability (Cohen’s κ ranges between .53 and .85). The accuracies of the automated analysis were on average 10% lower than the accuracies of the manual analysis. These findings enable reflection analytics that is immediate and scalable.

Based on a literature review, reflections in written text are rare. The reported proportions of reflection are based on different baselines, making comparisons difficult. In contrast, this research reports on the proportion of occurrences of elements of reflection based on sentence level. This metric allows to compare proportions of elements of reflection. Previous studies are based on courses tailored to foster reflection. The reported proportions represent more the success of a specific instruction than informing about proportions of reflections occurring in student writings in general. This study is based on a large sample of course forum posts of a virtual learning environment. In total 1000 sentences were randomly selected and manually classified according to six elements of reflection. Five raters rated each sentence. Agreement was calculated based on a majority vote. The proportions of elements of reflection are reported and its potential application for course analytics demonstrated. The results indicate that reflections in text are indeed rare, and that there are differences within elements of reflection.

Each year, students contribute tens of thousands of comments about their student experience via the Student Experience on a Module Survey (SEaM survey). There remains a challenge as to how best utilise this data effectively for understanding module performance and planning module revisions. This presentation takes a big data perspective analysing tens of thousands of comments of the OU wide administered SEaM survey. It uses automated empirical text analysis methods to detect hot topics students talk about during an academic year and it evaluates the sentiment that students express towards these topics. This presentation shows results from several lines of investigation that started in research about the automated detection of reflective keywords in writings (Ullmann, 2015c, 2017b)to its first feasibility study in the context of the Open University (QE PID 'Applicability of Natural Language Processing to analyse SEaM open comment data'), to its first application in the context of quality enhancement at the Open University, such as the SEaM comment analyses for WELS (Ullmann, 2015b, 2015a), in the context of widening access (Coughlan, Ullmann, & Lister, 2017), group tuition (Ullmann, 2017a), and the latest Data Wrangler Scholarly Insight Report Spring 2018.

Most distance learning institutions collect vast amounts of learning data. Making sense of this 'Big Data' can be a challenge, in particular when data are stored at different data warehouses and require advanced statistical skills to interpret complex patterns of data. As a leading institute on learning analytics, the Open University UK instigated in 2012 a Data Wrangling initiative. This provided every Faculty with a dedicated academic with expertise data analysis and whose task is to provide strategic, pedagogical and sense-making advice to staff and senior management. Given substantial changes within the OU (e.g. new Faculty structure, real-time dashboards, two large-scale adoptions of predictive analytics approaches, increased reliance on analytics), this embedded case study provides an in-depth review of lessons learned of five years of data wrangling. We will elaborate on the design of the new structure, its strengths and potential weaknesses, and affordances to be adopted by other institutions.

Accessibility cannot be fully achieved through adherence to technical guidelines, and must include processes that take account of the diverse contexts and needs of individuals. A complex yet important aspect of this is to understand and utilise feedback from disabled users of systems and services. Open comment feedback can complement other practices in providing rich data from user perspectives, but this presents challenges for analysis at scale. In this paper, we analyse a large dataset of open comment feedback from disabled students on their online and distance learning experience, and we explore opportunities and challenges in the analysis of this data. This includes the automated and manual analysis of content and themes, and the integration of information about the respondent alongside their feedback. Our analysis suggests that procedural themes, such as changes to the individual over time, and their experiences of interpersonal interactions, provide key examples of areas where feedback can lead to insight for the improvement of accessibility. Reflecting on this analysis in the context of our institution, we provide recommendations on the analysis of feedback data, and how feedback can be better embedded into organisational processes.

Despite their importance for educational practice, reflective writings are still manually analysed and assessed, posing a constraint on the use of this educational technique. Recently, research started to investigate automated approaches for analysing reflective writing. Foundational to many automated approaches is the knowledge of words that are important for the genre. This research presents keywords that are specific to several categories of a reflective writing model. These keywords have been derived from eight datasets, which contain several thousand instances using the log-likelihood method. Both performance measures, the accuracy and the Cohen's κ, for these keywords were estimated with ten-fold cross validation. The results reached an accuracy of 0.78 on average for all eight categories and a fair to good interrater reliability for most categories even though it did not make use of any sophisticated rule-based mechanisms or machine learning approaches. This research contributes to the development of automated reflective writing analytics that are based on data-driven empirical foundations.

The evidence shows that the use of learning analytics to improve and to innovate learning and teaching in Europe is still in its infancy. The high expectations have not yet been realised. Though early adopters are already taking a lead in research and development, the evidence on practice and successful implementation is still scarce. Furthermore, though the work across Europe on learning analytics is promising, it is currently fragmented. This underlines the need for a careful build-up of research and experimentation, with both practice and policies that have a unified European vision. Therefore, the study suggests that work is needed to make links between learning analytics, the beliefs and values that underpin this field, and European priority areas for education and training 2020. As a way of guiding the discussion about further development in this area, the Action List for Learning Analytics is proposed. The Action List for Learning Analytics focuses on seven areas of activity. It outlines a set of actions for educators, researchers, developers and policymakers in which learning analytics are used to drive work in Europe’s priority areas for education and training. Strategic work should take place to ensure that each area is covered, that there is no duplication of effort, that teams are working on all actions and that their work proceeds in parallel. Policy leadership and governance practices •Develop common visions of learning analytics that address strategic objectives and priorities •Develop a roadmap for learning analytics within Europe •Align learning analytics work with different sectors of education •Develop frameworks that enable the development of analytics •Assign responsibility for the development of learning analytics within Europe •Continuously work on reaching common understanding and developing new priorities Institutional leadership and governance practices •Create organisational structures to support the use of learning analytics and help educational leaders to implement these changes •Develop practices that are appropriate to different contexts •Develop and employ ethical standards, including data protection Collaboration and networking •Identify and build on work in related areas and other countries •Engage stakeholders throughout the process to create learning analytics that have useful features •Support collaboration with commercial organisations Teaching and learning practices •Develop learning analytics that makes good use of pedagogy •Align analytics with assessment practices Quality assessment and assurance practices •Develop a robust quality assurance process to ensure the validity and reliability of tools •Develop evaluation checklists for learning analytics tools Capacity building •Identify the skills required in different areas •Train and support researchers and developers to work in this field •Train and support educators to use analytics to support achievement Infrastructure •Develop technologies that enable development of analytics •Adapt and employ interoperability standards