Abstract Lexical semantic change (LSC) detection investigates changes in word meaning over time, focusing on language use at the lexical-semantic level from a diachronic perspective. The field has made significant progress over the past two decades, driven by the increased availability of multilingual benchmarks, notable performance improvements, and growing interdisciplinary applications. In this paper, we review the evolution of LSC models and benchmark constructions within the context of popular shared tasks. By categorizing LSC models into generations defined by key components, our investigation suggests that performance breakthroughs have been largely driven by advances in semantic representations , transitioning from count-based models to recent transformer-based approaches. Notably, transformer models have established themselves as state-of-the-art by integrating Word-in-Context tasks, which emphasize semantic proximity in context . Furthermore, we review substantial studies that primarily leverage diachronic word embeddings to explore political, social, and cultural contexts beyond the linguistic domain. Our work provides valuable insights for future model development and encourages further interdisciplinary exploration within digital humanities and social sciences.
Abstract Natural language processing (NLP) technologies increasingly shape public life, yet their deployment for social good remains unevenly distributed across domains, languages, and geographies. This piece inaugurates the NLP for Social Good column in this journal. In this piece, I map the current state of NLP for Social Good (NLP4SG) across nine application domains. The picture that emerges is one of striking imbalance: AI harms, inclusion, and digital violence attract the bulk of research attention, while poverty, peacebuilding, and environmental protection remain critically underexplored. I argue that the field must address three structural gaps, domain coverage, linguistic diversity, and evaluation methodology, if NLP is to fulfil its potential as a force for equitable social progress. The piece concludes with five directions that I believe will define the next chapter of NLP4SG research.
Abstract Living languages continuously evolve to reflect the cultural changes of human societies. This evolution manifests through neologisms (new words) or the semantic change of existing words (new meanings for existing words). Understanding the meaning of words is vital for interpreting texts from different cultures (regionalisms or slang), domains (e.g., technical terms), or time periods. In computer science, this phenomenon is relevant to computational linguistics tasks such as machine translation, information retrieval, and question answering. Semantic change can impact the performance of these applications, making it important to understand and characterize these changes formally. This problem has recently attracted significant attention from the computational linguistics community. Several approaches can detect semantic changes with good precision, but more effort is needed to characterize how word meanings change and to determine how to mitigate the impact of this phenomenon. This survey provides a comprehensive overview of existing approaches to the characterization of semantic change . We also formally define three classes of characterization: change in dimension (whether a word’s meaning becomes broader or narrower), change in orientation (whether a word acquires a more pejorative or ameliorative sense), and change in relation (whether a word is used in a new figurative context, such as a metaphor or metonymy). We demonstrate the applicability of this formalism on existing corpora, summarize the key aspects of selected publications, and discuss current needs and trends in research on semantic change characterization.
Abstract Natural language processing (NLP) has moved from a specialized research field into the everyday infrastructure of writing, search, translation, education, journalism, public administration, and scientific work. This transition changes what counts as progress. Accuracy, fluency, and benchmark performance remain important, but they are no longer sufficient when language technologies shape knowledge, decisions, identities, and public trust. This column introduces Responsible NLP as a research orientation that integrates fairness, transparency, privacy, safety, cultural diversity, environmental awareness, and human agency across the full life cycle of language technologies. It argues that responsibility is not an external constraint on innovation, but a condition for meaningful and trustworthy innovation. Future research must therefore ask not only whether an NLP system works but also for whom it works, under which assumptions, with what risks, and with what forms of accountability.
Semantic textual similarity (STS) is to measure semantic equivalence between sentences; it plays an important role in natural language processing (NLP) tasks. The major core of STS is text representation. This paper studies how to obtain a sentence skeleton for text representation in STS. Unlike most existing syntax models, we propose a skeleton-based reinforcement learning method to identify the skeleton and construct sentence representations. Parallel networks are adopted to extract features of different dimensions in the sentence. In the framework of parallel networks, two sentence representation models are designed: context constrained LSTM (CC-LSTM) and Adorned skeleton LSTM (AS-LSTM). CC-LSTM builds the sentence representation by constraining the word context. AS-LSTM constructs the sentence representation through using the identified skeleton and its qualifiers. Our approach achieves good results without using external resources. Especially AS-LSTM, which outperforms the state-of-the-art without using external resources in the SICK dataset.
Online counselling services have seen increased use in recent years, providing critical emergency mental health support. These interactions are typically long, complex, and varied in the dialogue between help seekers and counsellors. The lack of domain-specific models, especially in low-resource languages, poses a significant challenge for the automatic detection of suicide risk in online chat services for mental health support. To address this challenge, our approach adapts a general-purpose large language model (LLM) to the suicide prediction task that employs a two-stage classification architecture to deal with sparse and imbalanced data. It extends the state of the art by: (1) incorporating psychological theory into model training and (2) capturing key aspects of conversation structure in counselling sessions. We evaluate the performance of the proposed LLM against the state-of-the-art LLMs for suicide detection on thousands of conversations in the Hebrew language from a leading national online counselling service in Israel. Results show that the proposed LLM outperformed existing state-of-the-art approaches in detecting suicide risk, as measured by relevant literature metrics. Moreover, the LLM outperforms other approaches even in the early stages of a conversation, which is crucial for real-time detection in practice. We also discuss the ethical implications of combining LLMs in counselling services. The contributions of this work are (1) extending existing LLM architectures to incorporate domain-specific information; (2) evaluating LLM technologies in the context of socially relevant problems; and (3) introducing novel LLM tools for resource-constrained languages.
Twitter is one of the most widely used social networks globally but is also notorious for harbouring significant levels of hate speech. This paradox has garnered the attention of companies and governments concerned about how digital hate can fragment communities and incite real-world violence. While extensive research has focused on detecting hate speech, there is a lack of comprehensive analysis of the actors involved, their characteristics, and their interactions within the online environment. Common assumptions categorise users into merely two groups-haters and non-haters-overlooking the existence of other groups that may more accurately represent the dynamics of hate dissemination. Additionally, existing user classification models often rely on large volumes of tweet data, a limitation given the restricted access to the Twitter API. Social networks are also frequently visualised using graphs where edges represent only superficial relationships. This study addresses these gaps through five research questions. We propose formal user clustering methods and develop a classifier that uses exclusively profile attributes-information more readily obtainable from the network. We also introduce a more nuanced definition of interaction for graph edges, based on ideological support or opposition between users. To conduct our analysis, we have extracted two complementary datasets: (i) a keyword-based corpus of 3.3M Spanish tweets containing hate-associated terms, of which 1.6M unique tweets were retained after filtering and (ii) a user-based corpus comprising timelines of $\approx$ 3,000 users linked to hate speech, totalling over 3 M tweets. Our results reveal the existence of three primary user classes-haters, upstanders, and neutrals-in contrast to the conventional binary classification. We demonstrate that profile attributes are reliable indicators for automatically classifying users and find significant statistical differences between these classes. Finally, we develop a graph visualisation tool to assist authorities in analysing interactions among different user types, providing a useful exploratory tool to support the analysis of online hate and inform potential mitigation strategies.
Soft prompt learning methods offer parameter-efficient tuning of pre-trained language models for few-shot scenarios. This study explores the integration of supervised contrastive learning (SCL) into two leading soft prompt tuning models: DifferentiAble pRompT (DART) and PTuning. By incorporating SCL as an auxiliary task, we observe consistent performance enhancements across 13 few-shot natural language understanding tasks, including benchmarks such as SST-2, TREC, MNLI, and real-world datasets such as Overruling, TC, and ADE. We also delve into SCL's impact in label-imbalanced settings, introducing a novel approach called balanced batch in SCL (BBSCL). BBSCL employs balanced mini-batches, sampling the majority class proportionally to the minority class to stabilize SCL calculations. Our results indicate that SCL and BBSCL significantly boost the performance and robustness of soft prompt learning models, especially on datasets with intricate label spaces. Experimentally, DART + SCL and PTuning + SCL outperform their base models by an average of $2.1\%$ across the 13 tasks. Additionally, we find that SCL's contribution is more substantial in scenarios with complex and less separable label spaces. Compared to large language models such as GPT-3.5 and OpenChat, our enhanced soft prompt learning models with SCL and BBSCL extensions exhibit superior performance in both balanced and imbalanced few-shot settings. This research not only improves the effectiveness of few-shot tuning techniques but also deepens our understanding of this area.
Neural Machine Translation (NMT), a subfield of Natural Language Processing, has seen significant advancements with the emergence of transformer architectures and generative artificial intelligence, demonstrating remarkable performance in various languages. However, translating Arabic dialects remains a notable challenge that becomes very pronounced primarily due to their morphological complexity and divergence from standardised grammatical rules. In this paper, we present a hybrid approach for translating the Maghrebi dialects into/from Modern Standard Arabic (MSA). The approach takes advantage of the strengths of the transformer architecture and the BERT language model for transfer learning of representations. To achieve this, we incorporated BERT embeddings into the encoder and decoder stacks of the transformer architecture. The BERT architecture, which we utilised, was trained in a self-supervised manner on Maghrebi dialects and Arabic corpora. The resulting BLEU/BERTScore/ChrF/METEOR scores for the approach were 14.148/79.414/28.885/28.428 and 8.961/20.994/19.465 (BLEU/ChrF/METEOR) for the translation in both directions using the raw data, demonstrating competitive performance compared to ChatGPT and Gemini Large Language Models (LLMs). Furthermore, we evaluated the approach using an ablation study with fine-tuned NLLB-200 and against three combinations of tokeniser techniques used in conjunction with the transformer architecture: Byte-Pair Encoding (BPE) tokeniser, WordPiece tokeniser, and BERT tokeniser. Both evaluations, including human evaluation, confirm the efficacy of our method.
In the digital information age, artificial intelligence is increasingly being applied to national governance and judicial decision-making assistance. Existing studies lack case studies and empirical analyses of the effectiveness of large models in aiding judicial decisions. To address this research gap, this study designs a comprehensive evaluation framework encompassing five core task dimensions: Task-oriented Information Extraction, Legal Article Citation, Event Extraction, Judicial Decision Generation, and Legal Opinion Generation. By using carefully crafted prompts to activate the legal reasoning capabilities of the models, we conducted extensive testing on 13 mainstream large language models (LLMs). The experimental results demonstrate that large models perform excellently in processing legal texts and providing preliminary legal opinions, but still exhibit shortcomings in complex legal reasoning and precise decision-making. On this basis, we applied a weakly supervised learning strategy to fine-tune the LLMs for targeted improvements. The results indicate that introducing a small amount of task-specific learning can significantly enhance the performance of LLMs in judicial tasks. This further underscores the critical role of data and the acquisition of domain-specific knowledge in applying AI technology to judicial tasks. Additionally, this study briefly discusses the issue of the boundaries of AI's involvement in judicial activities, aiming to provide theoretical foundations and practical guidance for the deep integration of AI technology with legal practice.
The study of the attribution of input features to the output of neural network models is an active area of research. While numerous Explainable AI (XAI) techniques have been proposed to interpret these models, the systematic and automated evaluation of these methods in sequence-to-sequence (seq2seq) models is less explored. This paper introduces a new approach for evaluating explainability methods in transformer-based seq2seq models, building upon the forward simulation of XAI methods. We use teacher-derived attribution maps as a structured side signal to guide a student model, and quantify the utility of different attribution methods through the student's ability to simulate targets. Using the Inseq library, we extract attribution scores over source-target sequence pairs and inject these scores into the attention mechanism of a student transformer model under four composition operators (addition, multiplication, averaging, and replacement). Across three language pairs (de-en, fr-en, ar-en) and attributions from Marian-MT and mBART models, Attention, Value Zeroing, and Layer Gradient $ imes$ Activation consistently yield the largest gains in BLEU (and corresponding improvements in chrF) relative to baselines. In contrast, other gradient-based methods (Saliency, Integrated Gradients, DeepLIFT, Input $ imes$ Gradient, GradientShap) lead to smaller and less consistent improvements. These results suggest that different attribution methods capture distinct signals and that attention-derived attributions better capture the alignment between source and target representations in seq2seq models. Finally, we introduce an Attributor transformer that, given a source-target pair, learns to reconstruct the teacher's attribution map. Our findings demonstrate that the more accurately the Attributor can reproduce attribution maps, the more useful an injection of those maps is for the downstream task. The source code can be found on GitHub. a
Named Entity Recognition (NER) is a machine learning task that traditionally relies on supervised learning and annotated data. Acquiring such data is often a challenge, particularly in specialized fields like medical, legal, and financial sectors. Those are commonly referred to as low-resource domains, which comprise long-tail entities, due to the scarcity of available data. To address this, data augmentation techniques are increasingly being employed to generate additional training instances from the original dataset. In this study, we evaluate the effectiveness of two prominent text augmentation techniques, Mention Replacement and Contextual Word Replacement, on two widely-used NER models, Bi-LSTM+CRF and BERT. We conduct experiments on four datasets from low-resource domains, and we explore the impact of various combinations of training subset sizes and number of augmented examples. We not only confirm that data augmentation is particularly beneficial for smaller datasets, but we also demonstrate that there is no universally optimal number of augmented examples, i.e., NER practitioners must experiment with different quantities in order to fine-tune their projects.
Crawling parallel texts-texts that are mutual translations-from the Internet is usually done following a brute-force approach: documents are massively downloaded in an unguided process, and only a fraction of them end up leading to actual parallel content. In this work, we propose a smart crawling method that guides the crawl towards finding parallel content more rapidly. We follow a neural approach that consists in adapting a pre-trained multilingual language model based on the encoder of the Transformer architecture by fine-tuning it for two new tasks: inferring the language of a document from its Uniform Resource Locator (URL) and inferring whether a pair of URLs link to parallel documents. We evaluate both models in isolation and their integration into a crawling tool. The results demonstrate the individual effectiveness of both models and highlight that their combination enables us to address a practical engineering challenge: the early discovery of parallel content during web crawling in a given language pair. This leads to a reduction in the amount of downloaded documents deemed useless and yields a greater quantity of parallel documents compared to conventional crawling approaches.
