Abstract When you use a large language model (LLM) in your research, you often need to formulate a prompt to elicit some relevant output from the LLM. This step is challenging since (1) LLMs are known to be brittle and their results may vary drastically between different prompts; and (2) for any given task, there are infinitely many possible prompts. Thus we end up with the following problem: If you cannot try out infinitely many prompts, how do you justify your selected prompt? This article discusses different ways in which authors may motivate their choices, and urges authors to consider how their prompting strategy might be justified.
Abstract Reproductions in NLP/ML indicate worryingly low levels of reproducibility, but their results are hard to interpret, compare, and learn from, because they are reported in qualitative terms with varying, often unstated, criteria. We propose QRA, a quantitative approach to reproducibility assessment that (i) produces continuous-valued degree of reproducibility assessments at three levels of granularity; (ii) utilizes reproducibility measures that are comparable across different studies; and (iii) grounds expectations about degree of reproducibility in similarity between experiments. QRA facilitates more informative reproducibility assessments and conclusions about the causes of better/poorer reproducibility. We demonstrate its benefits by applying QRA to three sets of comparable experiments, obtaining clear evidence that degree of reproducibility depends on similarity of experiment properties, system type, and evaluation method.
Abstract Automatic methods for scoring text readability have been studied for over a century, and are widely used in research and in user-facing applications in many domains. Thus far, the development and evaluation of such methods have primarily relied on two types of offline human behavioral data, performance on reading comprehension tests and ratings of text readability levels. In this work, we instead focus on a fundamental and understudied aspect of readability, real-time reading ease, captured with online reading measures using eye tracking. We introduce a new cognitive evaluation framework for readability scoring methods that quantifies their ability to account for reading ease, while controlling for content variation across texts. Applying this evaluation to prominent traditional readability formulas, NLP-based methods, commercial systems used in education, and frontier LLMs suggests that they are all poor predictors of English reading ease in adults as compared to word properties commonly used in psycholinguistics for the prediction of reading times. This outcome holds across L1 and L2 speakers, different reading regimes, and textual units of different lengths. Our results reveal an important limitation of a wide range of methods for readability scoring, highlight the utility of real-time behavioral benchmarks for readability research, and call for new, cognitively driven readability scoring approaches that can better account for how humans experience texts in real time.
Abstract This study examines how language models access and recover linguistic information associated with the correct antecedents of two types of anaphoric elements: Anaphoric one and Noun Phrase Ellipsis (NPE), both of which require antecedent resolution for proper interpretation. In this study, we created manually annotated datasets (collectively referred to as AO-NPE datasets) from English Wikipedia, containing annotated instances of Anaphoric one and NPE. We assessed model performance by fine-tuning BERT and RoBERTa, and by evaluating GPT-4o mini, LLaMA 3.2, and Gemma-2 in a zero-shot setting. We conducted experiments using the forced-choice task and open-ended antecedent identification task on both humans and models with the same datasets. Notably, GPT-4o mini performed comparably to humans and outperformed other language models. However, results from both types of tasks revealed that language models performed better in identifying the antecedent for NPE than for Anaphoric one, a pattern opposite to that observed in human data. Linear distance and number agreement between antecedents and anaphoric elements also affected the performance, with language models more strongly influenced by these factors than humans. These findings suggest that the underlying mechanisms of anaphoric comprehension in language models may not fully align with those employed by humans: while humans flexibly integrate syntactic, semantic, and discourse information, LLMs rely more heavily on surface-level cues such as linear distance, number agreement, and syntactic parallelism. Additionally, model performance dropped significantly in the open-ended identification task compared to the forced-choice task, while human performance remained relatively stable. This further suggests potential limitations in anaphoric processing.
Abstract Recent reasoning-focused large language models (LLMs) (e.g., GPT-o1 and DeepSeek-R1) combine chain-of-thought prompting with reinforcement learning to generate explicit, multi-step derivations. However, their practical advantages over standard instruction-tuned models remain underexplored. In this article, we present a comprehensive empirical study to address these gaps. We compare reasoning-focused and non-reasoning LLMs across several critical dimensions: (1) their performance on mathematical and general reasoning benchmarks; (2) their generalization capabilities across 30 sub-disciplines spanning six scientific domains; (3) the qualitative and quantitative differences in their underlying solution processes; and (4) the impact of model compression via distillation on core reasoning capabilities. Crucially, to reliably conduct our in-depth process analysis, we address the critical issue of LLM-as-a-Judge reliability by proposing and validating a multi-level evaluation framework against human experts. Our experimental results demonstrate that reasoning-focused models significantly outperform non-reasoning counterparts on logic-intensive tasks such as mathematics and physics, while the latter remain competitive on factual or observational problems (e.g., biology, geography). Specifically, our validated process analysis reveals distinct styles: DeepSeek-R1 generates more elaborate reasoning processes, correlating with greater effectiveness on harder tasks, despite potential redundancy. In contrast, GPT-o1 exhibits greater conciseness and frequently presents solutions with superior clarity and efficiency, particularly on simpler problems, while struggling with complex scenarios. Additionally, reasoning capabilities can be partially preserved through distillation, with distilled reasoning models consistently outperforming their non-reasoning counterparts across all domains. These findings provide new insights into when and how explicit reasoning contributes to LLM performance, offering practical guidance for model development and deployment.
