Source code and data published at computational linguistics (*CL) venues are increasingly being shared via GitHub. While this generally is a favourable development for the accessibility and potential reusability of research artifacts in natural language processing (NLP), the long-term availability of such repositories has not been evaluated. In this squib, we discuss the availability of repositories linked in papers published in the Computational Linguistics (CL) journal as well as at ACL and its co-located events over the past ten years. Contrary to our expectations, we find that GitHub repositories linked in more recent ACL publications are unavailable at similar rates as in older publications, in parts due to an increase in empty and placeholder repositories. Similar trends hold for other *CL venues, but not for platforms other than GitHub.
Abstract When you use a large language model (LLM) in your research, you often need to formulate a prompt to elicit some relevant output from the LLM. This step is challenging since (1) LLMs are known to be brittle and their results may vary drastically between different prompts; and (2) for any given task, there are infinitely many possible prompts. Thus we end up with the following problem: If you cannot try out infinitely many prompts, how do you justify your selected prompt? This article discusses different ways in which authors may motivate their choices, and urges authors to consider how their prompting strategy might be justified.
Abstract Reproductions in NLP/ML indicate worryingly low levels of reproducibility, but their results are hard to interpret, compare, and learn from, because they are reported in qualitative terms with varying, often unstated, criteria. We propose QRA, a quantitative approach to reproducibility assessment that (i) produces continuous-valued degree of reproducibility assessments at three levels of granularity; (ii) utilizes reproducibility measures that are comparable across different studies; and (iii) grounds expectations about degree of reproducibility in similarity between experiments. QRA facilitates more informative reproducibility assessments and conclusions about the causes of better/poorer reproducibility. We demonstrate its benefits by applying QRA to three sets of comparable experiments, obtaining clear evidence that degree of reproducibility depends on similarity of experiment properties, system type, and evaluation method.
Traditional methods for studying human opinions often struggle to support representative and scalable research across countries. Large language models (LLMs) can serve as scalable proxies for simulating human opinions, enabling more efficient opinion analysis. However, this use of LLMs requires not only high average accuracy but also representational equality, that is, comparable simulation accuracy across populations. Uneven simulation accuracy may reproduce or amplify societal biases in downstream applications. This study systematically investigates country-level representational equality across 59 countries and finds substantial, systematic inequality. Populations from wealthier and more technologically advanced countries are simulated more accurately. We further compare two foundational intervention pathways, contextual adaptation and parametric modification, and show that improvements in average or target-group accuracy do not necessarily translate into greater representational equality. For contextual adaptation, native-language prompting generally improves accuracy but remains model-dependent, whereas additional information more often improves both accuracy and equality. For parametric modification, language-specific continued post-training improves accuracy for targeted language groups but unevenly, while preference alignment yields no systematic gains in accuracy or equality. Human-annotated preference data generally preserve accuracy better than AI-annotated data. These findings highlight the need for representational equality alongside accuracy and offer guidance for more inclusive, socially responsible LLM-based simulations.
