Background: Eye-tracking combined with machine learning (ML) has the potential to provide an objective and cost-effective method for identifying autism. However, current ML approaches that aim to identify Autism Spectrum Disorder (ASD) based on naturalistic web interactions have high variations on different pages and tasks. This inconsistency necessitates a more consistent approach since there is no clear pattern for identifying suitable web pages or tasks. In our previous study, we proposed using STA as a consistent approach, which demonstrated a higher mean accuracy than a relevant ML approach. However, our evaluation of STA was limited to a single dataset and a comparison with only one relevant ML approach. Objective: We aim to systematically evaluate the suitability of STA for autism identification using various datasets and compare its performance with other relevant ML approaches. Methods: We evaluated the accuracy of STA across various web pages, images, and tasks using three eye-tracking datasets and compared it to other ML approaches. Results: STA has demonstrated consistent accuracy across different web pages, images, and tasks, achieving a success rate of approximately 60%. STA has also outperformed other relevant ML approaches in terms of both the mean and standard deviation of accuracy rates on individual web pages, resulting in significantly higher results in most browsing tasks. Conclusion: STA is a consistent approach for identifying autism, as it achieves stable accuracy on various web pages, images, and tasks. However, further studies are necessary to improve its accuracy.

Recent developments in the use of large-language models have led to substantial improvements in the accuracy of content-based automated scoring of free-text responses. The reported accuracy levels suggest that automated systems could have widespread applicability in assessment. However, before they are used in operational testing, other aspects of their performance warrant examination. In this study, we explore the potential for examinees to inflate their scores by gaming the ACTA automated scoring system. We explore a range of strategies including responding with words selected from the item stem and responding with multiple answers. These responses would be easily identified as incorrect by a human rater but may result in false-positive classifications from an automated system. Our results show that the rate at which these strategies produce responses that are scored as correct varied across items and across strategies but that several vulnerabilities exist.

This paper presents an automated scoring approach for a formative assessment tool aimed at helping learner physicians enhance their communication skills through simulated patient interactions. The system evaluates transcribed learner responses by detecting key communicative behaviors, such as acknowledgment, empathy, and clarity. Built on an adapted version of the ACTA scoring framework, the model achieves a mean binary F1 score of 0.94 across 8 clinical scenarios. A central contribution of this work is the investigation of how to balance scoring accuracy with scalability. We demonstrate that synthetic training data offers a promising path toward reducing reliance on large, annotated datasets-making automated scoring more accurate and scalable.

Multiple-choice questions have become ubiquitous in educational measurement because the format allows for efficient and accurate scoring. Nonetheless, there remains continued interest in constructed-response formats. This interest has driven efforts to develop computer-based scoring procedures that can accurately and efficiently score these items. Early procedures were typically based on surface features of the responses or simple matching procedures, but recent developments in natural language processing have allowed for much more sophisticated approaches. This paper reports on a state-of-the-art methodology for scoring short answer questions supported by a large language model. Responses were collected in the context of a high-stakes test for medical students. More than 35,000 responses were collected across 71 studied items. Aggregated across all responses the proportion of agreement with human scores ranged from .97 to .99 (depending on specifics such as training sample size). In addition to reporting detailed results, the paper discusses practical issues that require consideration when adopting this type of scoring system.

In this investigation, we propose a solution for the author's gender identification task called AGI-P. This task has several real-world applications across different fields, such as marketing and advertising, forensic linguistics, sociology, recommendation systems, language processing, historical analysis, education, and language learning. We created a new dataset to evaluate our proposed method. The dataset is balanced in terms of gender using a random sampling method and consists of 1944 samples in total. We use accuracy as an evaluation measure and compare the performance of the proposed solution (AGI-P) against state-of-the-art machine learning classifiers and fine-tuned pre-trained multilingual language models such as DistilBERT, mBERT, XLM-RoBERTa, and Multilingual DEBERTa. In this regard, we also propose a customized fine-tuning strategy that improves the accuracy of the pre-trained language models for the author gender identification task. Our extensive experimental studies reveal that our solution (AGI-P) outperforms the well-known machine learning classifiers and fine-tuned pre-trained multilingual language models with an accuracy level of 92.03%. Moreover, the pre-trained multilingual language models, fine-tuned with the proposed customized strategy, outperform the fine-tuned pre-trained language models using an out-of-the-box fine-tuning strategy. The codebase and corpus can be accessed on our GitHub page at: https://github.com/mumairhassan/AGI-P

Scoring clinical patient notes (PNs) written by medical students is a necessary but resource-intensive task in medical education. This paper describes the organization and key lessons from a Kaggle competition on automated scoring of such notes. 1,471 teams took part in the competition and developed an extensive, publicly available code repository of varying solutions evaluated over the first public dataset for this task. The most successful approaches from this community effort are described and utilized in the development of a PN scoring system. We discuss the choice of models and system architecture with a view to operational use and scalability, and evaluate its performance on both the public Kaggle data (10 clinical cases, 43,985 PNs) and an extended internal dataset (178 clinical cases, 6,940 PNs). The results show that the system significantly outperforms a state-of-the-art existing tool for PN scoring and that task-adaptive pretraining using masked language modeling can be an effective approach even for small training samples.

This paper reports findings from the First Shared Task on Automated Prediction of Difficulty and Response Time for Multiple-Choice Questions. The task was organized as part of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA'24), held in conjunction with NAACL 2024, and called upon the research community to contribute solutions to the problem of modeling difficulty and response time for clinical multiple-choice questions (MCQs). A set of 667 previously used and now retired MCQs from the United States Medical Licensing Examination (USMLE (R)) and their corresponding difficulties and mean response times were made available for experimentation. A total of 17 teams submitted solutions and 12 teams submitted system report papers describing their approaches. This paper summarizes the findings from the shared task and analyzes the main approaches proposed by the participants.

In this paper, we evaluate and compare new variants of a popular rule-based anaphora resolution algorithm with the original version. We seek to establish data (always) outperform the original rule-based algorithm. The results of this study suggest that while algorithms based in Deep Learning and LLMs usually perform better than rule-based ones, this is not always the case, and we argue that rule-based approaches still have a place in today's research.

Deep learning (DL) has taken the research community by storm. The employment of DL techniques can be seen everywhere, DL has been pervasive as methodology, and the area of multiple-choice question generation is also joining the club of DL applications. In this chapter, we propose a DL methodology to solve a problem which to our knowledge has not been tackled before: the generation of multiple-choice questions (MCQs) which are based on the information of not one sentence only, but on a sequence of sentences.

The Holocaust was a tragic and catastrophic event in World War II (WWII) history that resulted in the loss of millions of lives. In recent years, the emergence of the field of digital humanities has made the study of Holocaust testimonies an important area of research for historians, Holocaust educators, social scientists, and linguists. One of the challenges in analysing Holocaust testimonies is the recognition and categorisation of named entities such as concentration camps, military officers, ships, and ghettos, due to the scarcity of annotated data. This paper presents a research study on a domain-specific hybrid named-entity recognition model, which focuses on developing NER models specifically tailored for the Holocaust domain. To overcome the problem of data scarcity, we employed hybrid annotation approach to training different transformer model architectures in order to recognise the named entities. Results show transformer models to have good performance compared to other approaches.

Relationship extraction from unstructured data remains one of the most challenging tasks in the field of Natural Language Processing (NLP). The complexity of relationship extraction arises from the need to comprehend the underlying semantics, syntactic structures, and contextual dependencies within the text. Unstructured data poses challenges with diverse linguistic patterns, implicit relationships, contextual nuances, complicating accurate relationship identification and extraction. The emergence of Large Language Models (LLMs), such as GPT (Generative Pre-trained Transformer), has indeed marked a significant advancement in the field of NLP. In this work, we assess and evaluate the effectiveness of LLMs in relationship extraction in the Holocaust testimonies within the context of the Historical realm. By delving into this domain-specific context, we aim to gain deeper insights into the performance and capabilities of LLMs in accurately capturing and extracting relationships within the Holocaust domain by developing a novel knowledge graph to visualise the relationships of the Holocaust. To the best of our knowledge, there is no existing study which discusses relationship extraction in Holocaust testimonies. The majority of current approaches for Information Extraction (IE) in historic documents are either manual or OCR based. Moreover, in this study, we found that the Subject-Object-Verb extraction using GPT3-based relations produced more meaningful results compared to the Semantic Role labeling-based triple extraction.

This chapter discusses the evolution of natural language processing (NLP) approaches to text representation and how different ways of representing text can be utilized for a relatively understudied task in educational assessment – that of predicting item characteristics from item text. The first part of the chapter gives an introductory overview of the transition from hypothesis-driven linguistic features to non-contextualized and then contextualized embeddings. This overview is intended for assessment professionals who do not have a background in NLP. The second part demonstrates how these approaches could be applied to predicting item difficulty, response time, and item biserial for a set of clinical multiple-choice questions from a high-stakes licensing exam. These items are written to a common reading level so that they differ only in the difficulty of the construct they measure (i.e., clinical knowledge). The chapter concludes by discussing practical considerations for developing such models (e.g., the role of training data), as well as the implications of model interpretability and the use of the predictions in the context of high-stakes assessment.

Automated detection of high-functioning autism in adults is a highly challenging and understudied problem. In search of a way to automatically detect the condition, we explore how eye-tracking data from reading tasks can be used. Our previous work shows that eye-tracking data from web page-processing tasks provide suitable signal for detecting high-functioning autism, achieving 75% accuracy for our sample. This research line leads us to ask further: how do reading passages and reading tasks compare to web pages and web-processing tasks for the purpose of autism detection. In this chapter, we hypothesize that the reading differences between adults with and without high-functioning autism captured through eye-tracking data that would be a useful marker for differentiating between the two groups. In this new study, we collect data from 20 reading passages and 60 comprehension questions associated to these passages organized into three separate data subsets. This data are used to train a variety of machine learning classifiers using various combinations of features and granularity levels for gaze data extraction. The best performing classifiers achieve accuracy of 57%, 83%, and 70% for the three subsets of data. These results are discussed in terms of their implications for autism detection and reading research as well as in comparison to similar studies using web pages and images as stimuli.

This paper presents the ACTA system, which performs automated short-answer grading in the domain of high-stakes medical exams. The system builds upon previous work on neural similarity-based grading approaches by applying these to the medical domain and utilizing contrastive learning as a means to optimize the similarity metric. ACTA is evaluated against three strong baselines and is developed in alignment with operational needs, where low-confidence responses are flagged for human review. Learning curves are explored to understand the effects of training data on performance. The results demonstrate that ACTA leads to substantially lower number of responses being flagged for human review, while maintaining high classification accuracy.

This paper presents a corpus of 43,985 clinical patient notes (PNs) written by 35,156 examinees during the high-stakes USMLE (R) Step 2 Clinical Skills examination. In this exam, examinees interact with standardized patients people trained to portray simulated scenarios called clinical cases. For each encounter, an examinee writes a PN, which is then scored by physician raters using a rubric of clinical concepts, expressions of which should be present in the PN. The corpus features PNs from 10 clinical cases, as well as the clinical concepts from the case rubrics. A subset of 2,840 PNs were annotated by 10 physician experts such that all 143 concepts from the case rubrics (e.g., shortness of breath) were mapped to 34,660 PN phrases (e.g., dyspnea, difficulty breathing).

Encapsulators are linguistic units which establish coherent referential connections to the preceding discourse in a text. In this paper, we address the challenge of automatically analysing the pronominal encapsulator ello in Spanish text. Our method identifies, for each occurrence, the antecedent of the pronoun (including its grammatical type), the connective phrase which combines with the pronoun to express a discourse relation linking the antecedent text segment to the following text segment, and the type of semantic relation expressed by the complex discourse marker formed by the connective phrase and pronoun. We describe our annotation of a corpus to inform the development of our method and to finetune an automatic analyser based on bidirectional encoder representation transformers. On testing our method, we find that it performs with greater accuracy than three baselines (0.76 for the resolution task), and sets a promising benchmark for the automatic annotation of occurrences of the pronoun ello, their antecedents, and the semantic relations between the two text segments linked by the connective in combination with the pronoun.