The widespread use of social networks and online platforms has contributed to the continued presence of hate speech and offensive language in digital spaces. Despite growing interest in automated content moderation, most academic research focuses on high-resource languages, such as English, German, or French, where manually labeled corpora are readily available; consequently, current approaches are limited in terms of generalizability for limited-resource language settings. To address this gap, we propose a novel Romanian dataset for offensive language detection - RO-Offense - with 12,445 manually annotated comments on sports-related articles for coarse-grained and fine-grained classification. All offensive messages were labeled according to their degree of offensiveness into Profanity, Insult, and Abuse. While BERT-based solutions still have the best performance in fine-grained multiclass classifications, we narrow this gap by releasing the ROC Guardian model family, a set of Granite Guardian v3-based LLMs fine-tuned on the RO-Offense dataset. These results further emphasize the importance of high-quality, task-specific datasets for advancing NLP research in lower-resourced languages.

The broad adoption of Large Language Models (LLMs) has increased the need for human-curated datasets that serve as evaluation benchmarks. This need is particularly pronounced for non-English languages and for tasks that are inherently subjective and require multiple human perspectives. One such example is the development of benchmarks designed to assess the cultural awareness of LLMs. Statistics and data science courses offer a potential setting for developing such benchmarks while teaching students to apply LLM evaluation techniques using statistical inference. This paper presents a pilot project in which students in a statistics course within a data science engineering program created culturally diverse multiple-choice questions, generated answers using LLMs, and applied statistical methods to assess model accuracy. Student feedback indicated the project was engaging and useful for learning, while also highlighting a notable reliance on LLMs, particularly for interpreting statistical results. The resulting dataset comprises 1355 multiple-choice questions across 18 categories, including language, social media, and politics. After filtering valid items, the dataset was used to evaluate both closed- and open-source LLMs. Results show that the Gemini (closed-source) and Qwen (open-source) model families achieved the best performance, with improvements linked to model size, reasoning capabilities, and access to search tools. The best closed-source model achieved an accuracy of 97.66%, whereas the best open-source model achieved an accuracy of 79.07%. Qualitative analyses of errors in the filtering procedure and model reasoning process point to possible explanations into the challenges LLMs face when handling culturally specific content. Furthermore, results support a cultural injection hypothesis, whereby cultural knowledge is embedded during pretraining and accessed through instruction tuning. Through this work, we aim to demonstrate how statistics and data science courses can provide productive contexts for developing open-source benchmarks for non-English languages while also enriching students’ learning experiences. The dataset is publicly available.

Despite having approximately 24 million native speakers, Romanian remains a low-resource language for automatic speech recognition (ASR), with few accurate and publicly available systems. To address this gap, this study explores the challenges of adapting modern speech recognition models, such as wav2vec 2.0 and Conformer, to Romanian. Our investigation is a comprehensive analysis of the two models, their capabilities to adapt to Romanian data, and the performance of the trained models. The research also focuses on unique attributes of the Romanian language, data collection techniques, including weakly supervised learning, and processing methodologies. Building on the previously introduced Echo dataset of 378 h, we release CRoWL (Crawled Romanian Weakly Labeled), a weakly supervised dataset of 9000 h created via automatic transcription. We obtain strong results that, to the best of our knowledge, are competitive with or exceed publicly reported results for Romanian under comparable open evaluation settings, with Conformer attaining 3.01% WER on Echo + CRoWL and wav2vec 2.0 reaching 4.04% (Echo) and 4.17% (Echo + CRoWL). In addition to the datasets, we also release our most capable models as open source, along with their training plans, thereby providing a solid foundation for researchers interested in languages with limited representation.

Agentic social networks expose autonomous agents to large volumes of potentially malicious content, including prompt injection, social engineering, and unsafe execution requests. Large Language Model (LLM) judges can detect such content, but applying them at platform scale is expensive. We study the extent to which lightweight embedding-based prefilters can reduce this cost while retaining most of the judge’s unsafe predictions. We annotate 10,000 Moltbook posts and comments with a frontier LLM judge, assigning a binary safety verdict, severity level, malicious intent taxonomy labels, and Open Worldwide Application Security Project (OWASP) AI risk codes. Of the LLM-annotated samples, 9.1% were unsafe at severity 3 or above. Two human annotators with high inter-rater agreement (Cohen’s κ = 0.823) exhibited moderate agreement between their adjudicated labels and the LLM ratings (κ = 0.578). We compare centroid-based cosine prefilters over 3 off-the-shelf encoders against a supervised trained classifier. Reducing the input window size raises average precision by up to 6.3 percentage points. At an operating point calibrated to 0.80 recall, the projected cost of scanning 787,226 messages falls from $7,085 by 56.0% (MiniLM-L12-v2), 62.6% (BGE-M3), and 74.0% (trained classifier), while MiniLM runs roughly 32 times faster than the classifier. Prefiltering can therefore make frontier-judge screening affordable at scale, although jailbreak content remains the hardest category to retrieve.

Vision-Language Models (VLMs) largely follow the text-only LLM trajectory, excelling on English benchmarks but sharply degrading on low-resource languages, where neither large-scale image-text corpora nor culturally grounded evaluations exist. We present a systematic study of building a language-specific VLM for Romanian, covering the full pipeline from data construction to architectural choices. We translate established English VLM training and evaluation corpora into Romanian, applying machine translation to textual annotations and to in-image text, preserving visual grounding while adapting the textual content. Using this data, we train and ablate a series of VLMs to isolate the contribution of (i) vision backbones of varying scale and pretraining, (ii) language backbones from multilingual to Romanian-adapted LLMs, and (iii) OCR-style image-text data. We further curate HoraVQA, a culturally native evaluation set grounded in Romanian everyday scenes. Romanian-adapted VLMs consistently outperform their same-sized counterparts and, across all evaluated benchmarks, even surpass models from the next larger size category.

Speaker diarization is a key component for multiple downstream speech technologies, including speech transcription, meeting analytics, and conversational understanding; however, Romanian lacks publicly established diarization resources and benchmarks. This paper evaluates cross-lingual transfer of diarization systems pretrained on predominantly English data, under a strict no-adaptation policy. We compare an end-to-end neural diarization approach (MSDD) and a traditional modular pipeline (segmentation + speaker embeddings + clustering), both used as is with pretrained components. To enable controlled analysis despite the lack of Romanian diarization datasets, we construct a synthetic Romanian conversational benchmark with explicit conditions on speaker count (2-5) and overlap regime (no overlap versus overlap). We report the diarization error rate (DER) and Jaccard error rate (JER) across all conditions, analyze sensitivity to overlap and the number of speakers, and provide an error-component breakdown to identify dominant failure modes. Across all conditions, the end-to-end system outperforms the pipeline (DER 0.140 versus 0.267; JER 0.152 versus 0.320). Performance degrades with overlap and with increasing speaker count in both paradigms, with speaker confusion dominating the additional error under overlap.

Generating multiple text sequences and refining them through feedback is essential for improving the quality of outputs in many NLP tasks. While Large Language Models can leverage iterative feedback during inference, smaller models often lack this capability due to limited capacity and the absence of suitable training paradigms. In this paper, we propose a novel Feedback-Aware Inference approach that enables iterative sequence generation with integration of feedback signals. Our method allows models to generate multiple sequences, incorporate feedback from previous iterations, and refine outputs accordingly. This approach dynamically adjusts to different quality metrics, making it adaptable to various contexts and objectives. We evaluate our approach on two distinct tasks: Answer Selection for Question Generation and Keyword Generation, arguing for its generalizability and effectiveness. Results show that our method outperforms strong baselines, maintaining high performance across iterations and achieving superior results even with smaller, open-source models.

Transformer-based language models often achieve strong results on mathematical reasoning benchmarks while remaining fragile on basic numerical understanding and arithmetic operations. A central limitation is that numbers are processed as symbolic tokens whose embeddings do not explicitly encode numerical value, leading to systematic errors. We introduce a value-aware numerical representation that augments standard tokenized inputs with a dedicated prefix token whose embedding is explicitly conditioned on the underlying numerical value. This mechanism injects magnitude information directly into the model's input space while remaining compatible with existing tokenizers and decoder-only Transformer architectures. Evaluation on arithmetic tasks shows that the proposed approach outperforms baselines across numerical formats, tasks, and operand lengths. These results indicate that explicitly encoding numerical value is an effective and efficient way to improve fundamental numerical robustness in language models.

IT help desks at large organizations face a high volume of recurrent, well-documented user requests that nevertheless require human-written replies, creating a persistent staff workload that is repetitive in content but non-trivial in tone and procedural correctness. We present FroLineR, short for Front-Line Response, a system that drafts the initial staff reply to such tickets in the login and account-activation category and integrates into a human-in-the-loop ticketing workflow on a Romanian-language ticketing platform. The generator is an unmodified instruct model augmented with retrieval from a small set of hand-curated guide documents, using a Romanian system prompt refined over several rounds of staff review. To evaluate and refine the prompt without manual labeling, we cluster the first user message of every historical thread with both BERTopic and Semantic Signal Separation (S3), score configurations along coherence and lexical-diversity axes, and extract a 200-message evaluation set from the winning model. Prompt convergence was certified by several rounds of manual review by support staff. The production system is quantized to Q4_K_M GGUF, served through llama-cpp-python behind a small Flask API, and deployed with GPU offloading on the target server, reducing end-to-end per-answer latency from approximately 830 s on the server’s CPU to roughly 61 s once layers are offloaded to the GPU, with no observable degradation in answer quality.

Cybersecurity news articles often report disclosed vulnerabilities, but their unstructured format limits automated vulnerability tracking. This paper introduces a vulnerability information extraction task that maps cybersecurity news articles to structured records covering affected software or hardware, software version, reporting entity, discovery date, vulnerability type, Common Vulnerabilities and Exposures (CVE) identifier, Common Vulnerability Scoring System (CVSS) score, impact, and patch information.We introduce and publicly release a corpus of 1,000 English-language articles annotated with 35,038 spans and validated through expert review. Using a chronological train/validation/test split, we compare encoder-based sequence labeling, supervised decoder-based structured generation, and few-shot in-context learning with large instruction-tuned language models. The results show complementary behavior across paradigms: domain-adapted encoders are most effective for compact technical spans, while decoder-based models are better aligned with normalized field-level records and longer descriptive fields. Notably, a fine-tuned 7B decoder achieved comparable aggregate performance to a substantially larger few-shot LLM (Δ ≤ 0.01 in macro- and micro- F1), indicating that task-specific supervision can offset model-scale advantages for fixed-schema extraction. Our findings emphasize the trade-offs between span-based extraction and structured generation for vulnerability-focused cybersecurity information extraction.

An effective method of teaching across disciplines is to provide examples of high-quality work. However, an example may be significantly different from a student's current work, making it challenging for them to emulate. An ideal learning demonstration is a counterfactual version of the student work, an improved version that is still similar to their own. Existing automated approaches for counterfactual text generation using Large Language Models (LLMs) result in domain-specific systems that are difficult to translate into practical applications. We present the Gumbel Machine, a flexible, modular approach to generating counterfactuals that leverages LLM instruction-following capabilities while encouraging similarity to a reference factual text. Central to our approach is a novel, controlled decoding algorithm, β-Hindsight control, which uses latent randomness as a tunable similarity control mechanism during counterfactual generation. Experiments on datasets of student writing, scored on various criteria, demonstrate the effectiveness of our approach at generating counterfactuals both rubric-consistent and similar to a reference.

Subword tokenization introduces a computational layer in language models where many distinct token sequences decode to the same surface form and preserve meaning, yet induce different internal computations. Despite this non-uniqueness, language models are typically trained using a single canonical longest-prefix tokenization. We formalize homotokens-alternative valid subword segmentations of the same lexical item-as a strictly meaning-preserving form of data augmentation. We introduce a lightweight training architecture that conditions canonical next-token prediction on sampled homotoken variants via an auxiliary causal encoder and block-causal cross-attention, without modifying the training objective or token interface. In data-constrained pretraining, homotoken augmentation consistently delays overfitting under repeated data exposure and improves generalization across diverse evaluation datasets. In multilingual fine-tuning, we find that the effectiveness of homotokens depends on tokenizer quality: gains are strongest when canonical tokens are highly compressed and diminish when the tokenizer already over-fragments the input. Overall, homotokens provide a simple and modular mechanism for inducing tokenization invariance in language models.

The rate at which fake news articles are written and disseminated on social media is alarming, posing a significant threat to both national security and individual well-being. This paper supports mitigating the propagation of fake news by empowering individuals to fact-check articles by comparing them to those from trustworthy news organizations on a platform that helps them find alternative sources and identify discrepancies between the articles. The backbone of the solution is a 3-staged processing pipeline that processes the article, fetches alternative sources, and generates the final similarity report. For fine-tuning the models in the pipeline, two new corpora were created, as no existing datasets were available for Romanian. One corpus consisted of synthetically generated search queries for a given article, whereas the second consisted of human annotations of pairs of news articles, labeled as either similar or not similar. Our pipeline, named CrossRead, enables users to easily compare sources and quickly fact-check articles while working reliably with articles in Romanian. The presented platform also constitutes an excellent base for a more feature-rich solution, with numerous improvements possible to assist its users in their search for the truth.

Despite their remarkable progress across diverse domains, Large Language Models (LLMs) consistently fail at simple character-level tasks, such as counting letters in words, due to a fundamental limitation: tokenization. In this work, we frame this limitation as a problem of low mutual information and analyze it in terms of concept emergence. Using a suite of 19 synthetic tasks that isolate character-level reasoning in a controlled setting, we show that such capabilities emerge suddenly and only late in training. We find that percolation-based models of concept emergence explain these patterns, suggesting that learning character composition is not fundamentally different from learning commonsense knowledge. To address this bottleneck, we propose a lightweight architectural modification that significantly improves character-level reasoning while preserving the inductive advantages of subword models. Together, our results bridge low-level perceptual gaps in tokenized LMs and provide a principled framework for understanding and mitigating their structural blind spots. We make our code publicly available.

Nowadays, grammatical error correction (GEC) has a significant role in writing since even native speakers often face challenges with proficient writing. This research is focused on developing a methodology to correct grammatical errors in the Romanian language, a less-resourced language for which there are currently no up-to-date GEC solutions. Our main contributions include an open-source synthetic dataset of 345,403 Romanian sentences, a manually curated dataset of 3054 social media comments, a two-phased GEC approach, and a comparison with several Romanian models, including RoMistral and RoLama3, but also LanguageTool, GPT-4o mini, and GPT-4o. We consider a synthetic dataset to finetune our models, while we rely on two real-life datasets with genuine human mistakes (i.e., CNA and RoComments) to evaluate performance. Building an artificial dataset was necessary because of the scarcity of real-life mistake datasets, whereas introducing RoComments, a new genuine dataset, is argued by the necessity to cover errors amongst native speakers encountered in social media comments. We also introduce a two-phased approach, where we first identify the location of erroneous tokens in the sentence; next, the erroneous tokens are replaced by an encoder–decoder model. Our approach achieved an F0.5 of 0.57 on CNA and 0.64 on RoComments, surpassing by a considerable margin LanguageTool as well as an end-to-end version based on Flan-T5 and mT0 in most setups. While our two-phased method did not outperform GPT-4o, arguably by its smaller size and language exposure, it obtained on-par results with GPT-4o mini and achieved higher performance than all Romanian LLMs.