Automating the classification of instructional strategies from a large-scale online tutorial dialogue corpus is indispensable to the design of dialogue-based intelligent tutoring systems. Despite many existing studies employing supervised machine learning (ML) models to automate the classification process, they concluded that building a well-performed ML model is nontrivial since the sample size is commonly limited for model training. Based on this fact, we posited that the model performance can be further optimized by the design of input features and the selection of informative instances for model training. By reviewing the existing dialogue-related research, we found that contextual information (i.e., the content of preceding utterances) was a potential feature in many text classification tasks but underexplored in the classification of an instructional strategy. In addition, training a well-performed ML model (e.g., deep neural network) to recognize instructional strategies usually requires a large amount of manually annotated data, which are labor intensive. To alleviate the demand of manual annotation, researchers proposed the use of statistical active learning methods to select informative instances, but this method was rarely used for recognizing the instructional strategy in online tutoring dialogue. Therefore, our study aimed to investigate the improvement of automating the classification of dialogue acts—a popular approach to the detection of instructional strategies—from two perspectives. First, we explored whether and to what extent the incorporation of contextual information can boost a model's prediction performance. Then, we investigated the extent to which the recent active learning methods can alleviate the labor-intensive issues in training the ML model for recognizing the instructional strategies. Our study showed that: 1) the ML models trained on the features that included the contextual information achieved better performance than that of the models excluding it; 2) the effectiveness of the contextual information decayed after the ML model achieved an optimal performance; and 3) compared with the random baseline, active learning methods can select informative samples from the training dataset to train ML models, which can alleviate the labor-intensive issues.

Researchers have demonstrated that dialogue-based intelligent tutoring systems (ITS) can be effective in assisting students in learning. However, little research has attempted to explore the necessity of equipping dialogue-based ITS with one of the most important capabilities of human tutors, that is, maintaining polite interactions with students, which is essential to provide students with a pleasant learning experience. In this study, we examined the role of politeness by analysing a large-scale real-world dataset consisting of over 14K online human-human tutorial dialogues. Specifically, we employed linguistic theories of politeness to characterise the politeness levels of tutor-student-generated utterances, investigated the correlation between the politeness levels of tutors' utterances and students' problem-solving performance and quantified the power of politeness in predicting students' problem-solving performance by applying Gradient Tree Boosting. The study results showed that: (i) in the effective tutorial sessions (ie, sessions in which students successfully solved problems), tutors tended to be very polite at the start of a tutorial session and become more direct to guide students as the session progressed; (ii) students with better performance in solving problems tended to be more polite at the beginning and the end of a tutorial session than their counterparts who failed to solve problems; (iii) the correlation between tutors' polite expressions and students' performance was not evident in non-instructional communication; and (iv) politeness alone cannot adequately reveal students' problem-solving performance, and thus other factors (eg, sentiment contained in utterances) should also be taken into account.

Students and administrators can benefit from new analytics.

Dialogue acts (DAs) can represent conversational actions of tutors or students that take place during tutoring dialogues. Automating the identification of DAs in tutoring dialogues is significant to the design of dialogue-based intelligent tutoring systems. Many prior studies employ machine learning models to classify DAs in tutoring dialogues and invest much effort to optimize the classification accuracy by using limited amounts of training data (i.e., low-resource data scenario). However, beyond the classification accuracy, the robustness of the classifier is also important, which can reflect the capability of the classifier on learning the patterns from different class distributions. We note that many prior studies on classifying educational DAs employ cross entropy (CE) loss to optimize DA classifiers on low-resource data with imbalanced DA distribution. The DA classifiers in these studies tend to prioritize accuracy on the majority class at the expense of the minority class which might not be robust to the data with imbalanced ratios of different DA classes. To optimize the robustness of classifiers on imbalanced class distributions, we propose to optimize the performance of the DA classifier by maximizing the area under the ROC curve (AUC) score (i.e., AUC maximization). Through extensive experiments, our study provides evidence that (i) by maximizing AUC in the training process, the DA classifier achieves significant performance improvement compared to the CE approach under low-resource data, and (ii) AUC maximization approaches can improve the robustness of the DA classifier under different class imbalance ratios.

Dialogue Acts (DAs) can be used to explain what expert tutors do and what students know during the tutoring process. Most empirical studies adopt the random sampling method to obtain sentence samples for manual annotation of DAs, which are then used to train DA classifiers. However, these studies have paid little attention to sample informativeness, which can reflect the information quantity of the selected samples and inform the extent to which a classifier can learn patterns. Notably, the informativeness level may vary among the samples and the classifier might only need a small amount of low informative samples to learn the patterns. Random sampling may overlook sample informativeness, which consumes human labelling costs and contributes less to training the classifiers. As an alternative, researchers suggest employing statistical sampling methods of Active Learning (AL) to identify the informative samples for training the classifiers. However, the use of AL methods in educational DA classification tasks is under-explored. In this paper, we examine the informativeness of annotated sentence samples. Then, the study investigates how the AL methods can select informative samples to support DA classifiers in the AL sampling process. The results reveal that most annotated sentences present low informativeness in the training dataset and the patterns of these sentences can be easily captured by the DA classifier. We also demonstrate how AL methods can reduce the cost of manual annotation in the AL sampling process.

Abstract Overcoming vaccine hesitancy is critical to containing the COVID-19 pandemic in the United States. To increase vaccination rates, the State of Ohio launched a million dollar lottery in May 2021. Following a pre-registered analysis, we estimate the effects of Ohio’s lottery program Vax-a-Million on COVID-19 vaccination rates by comparing it to a “synthetic control” composed of eight other states. We find a statistically insignificant 1.3% decrease in the full vaccination rate in Ohio at the end of the lottery period. We investigate the robustness of our conclusion to model specifications through a multiverse analysis of 216 possible models, including longer time periods and alternative vaccination measures. The majority (88%) find small negative effects in line with the results of our pre-registered model. While our results are most consistent with a decrease in vaccination rate, they do not allow a firm conclusion on whether the lottery increased or decreased vaccine uptake.

To construct dialogue-based Intelligent Tutoring Systems (ITS) with sufficient pedagogical expertise, a trendy research method is to mine large-scale data collected by existing dialogue-based ITS or generated between human tutors and students to discover effective tutoring strategies. However, most of the existing research has mainly focused on the analysis of successful tutorial dialogue. We argue that, to better inform the design of dialogue-based ITS, it is also important to analyse unsuccessful tutorial dialogues and gain a better understanding of the reasons behind those failures. Therefore, our study aimed to identify effective tutoring strategies by mining a large-scale dataset of both successful and unsuccessful human–human online tutorial dialogues, and further used these tutoring strategies for predicting students' problem-solving performance. Specifically, the study adopted a widely-used educational dialogue act scheme to describe the action behind utterances made by a tutor/student in the broader context of a tutorial dialogue (e.g., asking/answering a question, providing hints). Frequent dialogue acts were identified and analysed by taking into account the prior progress that a student had made before the start of a tutorial session and the problem-solving performance the student achieved after the end of the session. Besides, we performed a sequence analysis on the inferred actions to identify prominent patterns that were closely related to students' problem-solving performance. These prominent patterns could shed light on the frequent strategies used by tutors. Lastly, we measured the power of these tutorial actions in predicting students' problem-solving performance by applying a well-established machine learning method, Gradient Tree Boosting (GTB). Through extensive analysis and evaluations, we identified a set of different action patterns that were pertinent to tutors and students across dialogues of different traits, e.g., students without prior progress in solving problems, compared to those with prior progress, were likely to receive more thought-provoking questions from their tutors. More importantly, we demonstrated that the actions taken by students and tutors during a tutorial process could not adequately predict student performance and should be considered together with other relevant factors (e.g., the informativeness of the utterances).

Education has faced unprecedented disruption during the COVID pandemic. Understanding how students have adapted as we have entered a different phase of the pandemic and some communities have returned to more typical schooling will inform a suite of policy interventions and subsequent research. We use data from an oral reading fluency (ORF) assessment—a rapid assessment taking only a few minutes that measures a fundamental reading skill—to examine COVID’s effects on children’s reading ability during the pandemic. We find that students in the first 200 days of the 2020–2021 school year tended to experience slower growth in ORF relative to prepandemic years. We also observe slower growth in districts with a high percentage of English language learners and/or students eligible for free and reduced-price lunch. These findings offer valuable insight into the effects of COVID on one of the most fundamental skills taught to children.

Existing research indicates that students prefer to work with tutors who express politely in online human-human tutoring, but excessive polite expressions might lower tutoring efficacy. However, there is a shortage of understanding about the use of politeness in online tutoring and the extent to which the politeness of instructional strategies can contribute to students’ achievement. To address these gaps, we conducted a study on a large-scale dataset (5,165 students and 116 qualified tutors in 18,203 online tutoring sessions) of both effective and ineffective human-human online tutorial dialogues. The study made use of a well-known dialogue act coding scheme to identify instructional strategies, relied on the linguistic politeness theory to analyse the politeness levels of the tutors’ instructional strategies, and utilised Gradient Tree Boosting to evaluate the predictive power of these politeness levels in revealing students’ problem-solving performance. The results demonstrated that human tutors used both polite and non-polite expressions in the instructional strategies. Tutors were inclined to express politely in the strategy of providing positive feedback but less politely while providing negative feedback and asking questions to evaluate students’ understanding. Compared to the students with prior progress, tutors provided more polite open questions to the students without prior progress but less polite corrective feedback. Importantly, we showed that, compared to previous research, the accuracy of predicting student problem-solving performance can be improved by incorporating politeness levels of instructional strategies with other documented predictors (e.g., the sentiment of the utterances).

Committing to a major is a fateful step in an undergraduate education, yet the relationship between courses taken early in an academic career and ultimate major issuance remains little studied at scale. Using transcript data capturing the academic careers of 26,892 undergraduates enrolled at a private university between 2000 and 2020, we describe enrollment histories by using natural-language methods and vector embeddings to forecast terminal major on the basis of course sequences beginning at college entry. We find that (a) a student’s very first enrolled course predicts their major 30 times better than random guessing and more than one-third better than majority-class voting, (b) modeling strategies substantially influence forecasting metrics, and (c) course portfolios vary substantially within majors, such that students with the same major exhibit relatively modest overlap.

Committing to a major is a fateful step in an undergraduate's education, yet the relationship between courses taken early in an academic career and ultimate major choice remains little studied at scale. We analyze transcript data capturing the academic careers of 26,892 undergraduates at a private university between 2000 and 2020. We forecast students' terminal major on the basis of course-choice sequences beginning at university entry. We represent course enrollment history using natural-language methods and vector embeddings. We find that a student's very first enrolled course predicts their terminal major thirty times better than random guessing and more than a third better than majority class voting.

Education has faced unprecedented disruption during the COVID-19 pandemic; evidence about the subsequent effect on children is of crucial importance. We use data from an oral reading fluency (ORF) assessment—a rapid assessment taking only a few minutes that measures a fundamental reading skill—to examine COVID’s effects on children’s reading ability during the pandemic in more than 100 U.S. school districts. Effects were pronounced, especially for Grades 2–3, but distinct across spring and fall 2020. While many students were not assessed in spring 2020, those who were seemed to have experienced relatively limited or no growth in ORF relative to gains observed in other years. In fall 2020, a far more representative set of students was observed. For those students, growth was more pronounced and seemed to approach levels observed in previous years. Worryingly, there were also signs of stratification such that students in lower-achieving districts may be falling further behind. However, at the level of individual students, those who were struggling with reading prior to the pandemic were not disproportionately impacted in terms of ORF growth. This data offers an important window onto how a foundational skill is being affected by COVID-19 and this approach can be used in the future to examine how student abilities recover as education enters a post-COVID paradigm.

College courses are often organized into hierarchical sequences, with foundational courses recommended or required as prerequisites for other offerings. While the wisdom of particular sequences is usually ascertained on the basis of faculty experience or student peer networks, machine learning techniques and ubiquitous transcript data make it possible to systematically identify the courses that best predict subsequent high achievement across entire curricula and student populations. We demonstrate the utility of this approach by analyzing five years of course sequences and earned grades for 13,218 undergraduates enrolled in courses with substantial quantitative content at a private research university. Findings indicate that prior completion of specific courses is positively associated with success in subsequent target courses, and suggest that academic planning could be enhanced through scaled observation of the revealed benefits of course sequences.

We explore the relationship between video playback speed and student learning outcomes. Using an experimental design, we present the results of a pre-registered study that assigns users to watch videos at either 1.0x or 1.25x speed. We find that students who consume sped content are more likely to get better grades in a course, attempt more content, and obtain more certificates. We also find that when videos are sped up, students spend less time consuming videos and are marginally more likely to complete more video content. These findings suggest that future study of playback speed as a tool for optimizing video content for MOOCs is warranted. Applications for reinforcement learning and adaptive content are discussed.

Student course reviews are rarely considered as research instruments, yet their ubiquity makes them potentially powerful tools for education data science. To illustrate this potential we utilize a corpus of 11,255 reviews of computer science courses submitted by students at a private research university to observe how students appraise their own learning and give advice to future students. We recover evidence of gendered self-perceptions and care-giving strategies in Computer Science courses. Among submitters: females understate their achievement of learning goals relative to males earning the same grades; females offer lengthier written advice to future students than males; advice written by females exhibits more positive tone. Findings affirm the potential of leveraging course reviews for archival, survey, and quasi-experimental research investigations going forward

We explore the relationship between video playback speed and student learning outcomes. Using an experimental design, we present the results of a pre-registered study that assigns users to watch videos at either 1.0x or 1.25x speed. We find that students who consume sped content are more likely to get better grades in a course, attempt more content, and obtain more certificates. We also find that when videos are sped up, students spend less time consuming videos and are marginally more likely to complete more video content. These findings suggest that future study of playback speed as a tool for optimizing video content for MOOCs is warranted. Applications for reinforcement learning and adaptive content are discussed.

Background/Context Early warning indicators (EWI) are often used by states and districts to identify students who are not on track to finish high school, and provide supports/interventions to increase the odds the student will graduate. While EWI are diverse in terms of the academic behaviors they capture, research suggests that indicators like course failures, chronic absenteeism, and suspensions can help identify students in need of additional supports. In parallel with the expansion of administrative data that have made early versions of EWI possible, new machine learning methods have been developed. These methods are data-driven and often designed to sift through thousands of variables with the purpose of identifying the best predictors of a given outcome. While applications of machine learning techniques to identify students at-risk of high school dropout have obvious appeal, few studies consider the benefits and limitations of applying those models in an EWI context, especially as they relate to questions of fairness and equity. Focus of Study In this study, we will provide applied examples of how machine learning can be used to support EWI selection. The purpose is to articulate the broad risks and benefits of using machine learning methods to identify students who may be at risk of dropping out. We focus on dropping out given its salience in the EWI literature, but also anticipate generating insights that will be germane to EWI used for a variety of outcomes. Research Design We explore these issues by using several hypothetical examples of how ML techniques might be used to identify EWI. For example, we show results from decision tree algorithms used to identify predictors of dropout that use simulated data. Conclusions/Recommendations Generally, we argue that machine learning techniques have several potential benefits in the EWI context. For example, some related methods can help create clear decision rules for which students are a dropout risk, and their predictive accuracy can be higher than for more traditional, regression-based models. At the same time, these methods often require additional statistical and data management expertise to be used appropriately. Further, the black-box nature of machine learning algorithms could invite their users to interpret results through the lens of preexisting biases about students and educational settings.

Student course reviews are rarely considered as research instruments, yet their ubiquity makes them promising tools for education data science. To illustrate this potential, we use a corpus of student reviews to observe gender differences in how students appraise their own learning and in the advice they give to to future students. We find systematic differences in who submits course reviews, with female and academically high-achieving students more likely to submit. Among submitters, we find (a) females understate their achievement of learning goals relative to males earning the same grades; (b) females offer lengthier written advice to future students than males; (c) advice written by females exhibits more positive tone, even after accounting for grades and course selections.

This study aims to investigate the role of politeness in online-tutoring practices by analyzing a large-scale human-human tutorial dialogue dataset. To this end, we employed linguistic theories of politeness to identify the politeness strategies contained in utterances made by tutors and students, and these strategies were further combined to quantify the politeness levels of tutors and students in a tutorial session. The results revealed that tutors had a similar level of politeness at the beginning of all dialogues, while students were more polite at the end if they successfully solved problems.

This research evaluates the psychometric properties of short-answer response items under a variety of grading rules in the context of a mobile learning platform in Africa.This work has three main findings.First, we introduce the concept of a differential device function (DDF), a type of differential item function that stems from the device a student uses to take an assessment.Second, we identify a plausible mechanism for this DDF by examining the keystroke requirements of smartphone and basic mobile phone users.We identify a set of platform design rules to mitigates this bias.Lastly, we suggest that the edit distance of student responses can be used as a tuning parameter to optimize the Cronbach's alpha of the assessment.We find that literal string evaluation performs poorly compared to other grading rules.Partial string matching with an edit distance of two provides the highest reliability across exams.This is a simple yet effective rule, which performs well across a variety of assessments.