We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data. These datasets are derived from web crawls from different sources and accompanied with a complete, open-source pipeline for document selection from web archives, text extraction from HTML, language identification for noisy texts, exact and near-deduplication, annotation with, among others, register labels, text quality estimates, and personally identifiable information; and final selection and filtering. We report on data quality probes through contrastive and analytical statistics, through manual inspection of samples for 24 languages, and through end-to-end evaluation of various language model architectures trained on this data. For multilingual LLM evaluation, we provide a comprehensive collection of benchmarks for nine European languages, with special emphasis on natively created tasks, mechanisms to mitigate prompt sensitivity, and refined normalization and aggregation of scores. Additionally, we train and evaluate a family of 57 monolingual encoder-decoder models, as well as a handful of monolingual GPT-like reference models. Besides the monolingual data and models, we also present a very large collection of parallel texts automatically mined from this data, together with a novel parallel corpus synthesized via machine translation.

Language identification (LID) is an essential step in building high-quality multilingual datasets from web data. Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related languages and to distinguish valid natural language from noise, which contaminates language-specific subsets, especially for low-resource languages. In this work we extend the OpenLID classifier by adding more training data, merging problematic language variant clusters, and introducing a special label for marking noise. We call this extended system OpenLID-v3 and evaluate it against GlotLID on multiple benchmarks. During development, we focus on three groups of closely related languages (Bosnian, Croatian, and Serbian; Romance varieties of Northern Italy and Southern France; and Scandinavian languages) and contribute new evaluation datasets where existing ones are inadequate. We find that ensemble approaches improve precision but also substantially reduce coverage for low-resource languages. OpenLID-v3 is available on https://huggingface.co/HPLT/OpenLID-v3.

Training large language models requires vast amounts of data, posing a challenge for less widely spoken languages like Norwegian and even more so for truly low-resource languages like Northern Sámi. To address this issue, we present a novel three-stage continual training approach that substantially improves the downstream performance together with the inference efficiency for the target languages. Based on our findings, we train, evaluate, and openly release a new generative language model for Norwegian Bokmål, Nynorsk, and Northern Sámi with 11.4 billion parameters: NorMistral-11B.

Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.

We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls from the Internet Archive. We describe our methods for data acquisition, management and processing of large corpora, which rely on open-source software tools and high-performance computing. Our monolingual collection focuses on low- to medium-resourced languages and covers 75 languages and a total of ~5.6 trillion word tokens de-duplicated on the document level. Our English-centric parallel corpus is derived from its monolingual counterpart and covers 18 language pairs and more than 96 million aligned sentence pairs with roughly 1.4 billion English tokens. The HPLT language resources are one of the largest open text corpora ever released, providing a great resource for language modeling and machine translation training. We publicly release the corpora, the software, and the tools used in this work.

This paper demonstrates how a graph-based semantic parser can be applied to the task of structured sentiment analysis, directly predicting sentiment graphs from text. We advance the state of the art on 4 out of 5 standard benchmark sets. We release the source code, models and predictions.

We discuss methodological choices in diagnostic evaluation and error analysis in meaning representation parsing (MRP), i.e. mapping from natural language utterances to graph-based encodings of semantic structure. We expand on a pilot quantitative study in contrastive diagnostic evaluation, inspired by earlier work in syntactic dependency parsing, and propose a novel methodology for qualitative error analysis. This two-pronged study is performed using a selection of submissions, data, and evaluation tools featured in the 2019 shared task on MRP. Our aim is to devise methods for identifying strengths and weaknesses in different broad families of parsing techniques, as well as investigating the relations between specific parsing approaches, different meaning representation frameworks, and individual linguistic phenomena-by identifying and comparing common error patterns. Our preliminary empirical results suggest that the proposed methodologies can be meaningfully applied to parsing into graph-structured target representations, as a side-effect uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches.

Structured sentiment analysis attempts to extract full opinion tuples from a text, but over time this task has been subdivided into smaller and smaller sub-tasks, e,g,, target extraction or targeted polarity classification. We argue that this division has become counterproductive and propose a new unified framework to remedy the situation. We cast the structured sentiment problem as dependency graph parsing, where the nodes are spans of sentiment holders, targets and expressions, and the arcs are the relations between them. We perform experiments on five datasets in four languages (English, Norwegian, Basque, and Catalan) and show that this approach leads to strong improvements over state-of-the-art baselines. Our analysis shows that refining the sentiment graphs with syntactic dependency information further improves results.

Detecting the argument components Claim and Premise is a central task in argumentation mining. Working with two annotated corpora from the genre of short argumentative texts, we extend a BiLSTM-CRF neural tagger to identify argumentative units and to classify their type (claim vs. premise). For the corpora we use, Persuasive Essays and Argumentative Microtexts, current methods relied on pre-computed non-contextual word embeddings such as Glove. In this paper, we adopt contextual word embeddings (Bert, RoBerta) and cast the problem as a sequence labeling task. We show that this step improves the state of the art for the Persuasive Essays, and we present strong initial results on applying the same approach to the Argumentative Microtexts.

RFC 9000, published in May 2021, marks an important milestone for the Internet's standardization body, the Internet Engineering Task Force (IETF): finally, the specification of the QUIC protocol is available. QUIC is the result of a five-year effort-and it is also the second of two major protocols (the first being SPDY, which later became HTTP/2) that Google LLC first deployed, and then brought to the IETF for standardization. This begs the question: when big players follow such a "shoot first, discuss later" approach, is IETF collaboration still "real", or is the IETF now being (mis-)used to approve protocols for standardization when they are already practically established, without really actively involving anyone but the main proponents?

We present the ongoing NorLM initiative to support the creation and use of very large contextualised language models for Norwegian (and in principle other Nordic languages), including a ready-to-use software environment, as well as an experience report for data preparation and training. This paper introduces the first large-scale monolingual language models for Norwegian, based on both the ELMo and BERT frameworks. In addition to detailing the training process, we present contrastive benchmark results on a suite of NLP tasks for Norwegian. For additional background and access to the data, models, and software, please see http://norlm.nlpl.eu

In this paper we present Uniform Meaning Representation (UMR), a meaning representation designed to annotate the semantic content of a text. UMR is primarily based on Abstract Meaning Representation (AMR), an annotation framework initially designed for English, but also draws from other meaning representations. UMR extends AMR to other languages, particularly morphologically complex, low-resource languages. UMR also adds features to AMR that are critical to semantic interpretation and enhances AMR by proposing a companion document-level representation that captures linguistic phenomena such as coreference as well as temporal and modal dependencies that potentially go beyond sentence boundaries.

Stephan Oepen, Omri Abend, Lasha Abzianidze, Johan Bos, Jan Hajic, Daniel Hershcovich, Bin Li, Tim O'Gorman, Nianwen Xue, Daniel Zeman. Proceedings of the CoNLL 2020 Shared Task: Cross-Framework Meaning Representation Parsing. 2020.

We discuss methodological choices in contrastive and diagnostic evaluation in meaning representation parsing, i.e. mapping from natural language utterances to graph-based encodings of semantic structure. Drawing inspiration from earlier work in syntactic dependency parsing, we transfer and refine several quantitative diagnosis techniques for use in the context of the 2019 shared task on Meaning Representation Parsing (MRP). As in parsing proper, moving evaluation from simple rooted trees to general graphs brings along its own range of challenges. Specifically, we seek to begin to shed light on relative strenghts and weaknesses in different broad families of parsing techniques. In addition to these theoretical reflections, we conduct a pilot experiment on a selection of top-performing MRP systems and two of the five meaning representation frameworks in the shared task. Empirical results suggest that the proposed methodology can be meaningfully applied to parsing into graph-structured target representations, uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches.

We present a neural end-to-end architecture for negation resolution based on a formulation of the task as a graph parsing problem. Our approach allows for the straightforward inclusion of many types of graph-structured features without the need for representation-specific heuristics. In our experiments, we specifically gauge the usefulness of syntactic information for negation resolution. Despite the conceptual simplicity of our architecture, we achieve state-of-the-art results on the Conan Doyle benchmark dataset, including a new top result for our best model.

Discourse Representation Theory (DRT) is a formal account for representing the meaning of natural language discourse. Meaning in DRT is modeled via a Discourse Representation Structure (DRS), a meaning representation with a model-theoretic interpretation, which is usually depicted as nested boxes. In contrast, a directed labeled graph is a common data structure used to encode semantics of natural language texts. The paper describes the procedure of dressing up DRSs as directed labeled graphs to include DRT as a new framework in the 2020 shared task on Cross-Framework and Cross-Lingual Meaning Representation Parsing. Since one of the goals of the shared task is to encourage unified models for several semantic graph frameworks, the conversion procedure was biased towards making the DRT graph framework somewhat similar to other graph-based meaning representation frameworks.

This tutorial is on representing and processing sentence meaning in the form of labeled directed graphs. The tutorial will (a) briefly review relevant background in formal and linguistic semantics; (b) semi-formally define a unified abstract view on different flavors of semantic graphs and associated terminology; (c) survey common frameworks for graph-based meaning representation and available graph banks; and (d) offer a technical overview of a representative selection of different parsing approaches. 1 Tutorial Content and Relevance All things semantic have been receiving heightened attention in recent years. Despite remarkable advances in vector-based (continuous, dense, and distributed) encodings of meaning, ‘classic’ (hierarchically structured and discrete) semantic representations continue to play an important role in ‘making sense’ of natural language. While parsing has long been dominated by tree-structured target representations, there is now growing interest in general graphs as more expressive and arguably more adequate target structures for sentence-level grammatical analysis beyond surface syntax and in particular for the representation of semantic structure. Today, the landscape of meaning representation approaches, annotated graph banks, and parsing techniques into these structures is complex and diverse. Graph-based semantic parsing has been a task in almost every Semantic Evaluation (SemEval) exercise since 2014. These shared tasks were based on a variety of different corpora with graph-based meaning annotations (graph banks), which differ both in their formal properties and in the facets of meaning they aim to represent. The goal of this tutorial is to clarify this landscape for our research community by providing a unifying view on these graph banks and their associated parsing problems, while working out similarities and differences between common frameworks and techniques. Based on common-sense linguistic and formal dimensions established in its first part, the tutorial will provide a coherent, systematized overview of this field. Participants will be enabled to identify genuine content differences between frameworks as well as to tease apart more superficial variation, for example in terminology or packaging. Furthermore, major current processing techniques for semantic graphs will be reviewed against a highlevel inventory of families of approaches. This part of the tutorial will emphasize reflections on codependencies with specific graph flavors or frameworks, on worst-case and typical time and space complexity, as well as on what guarantees (if any) are obtained on the wellformedness and correctness of output structures. Kate and Wong (2010) suggest a definition of semantic parsing as “the task of mapping natural language sentences into complete formal meaning representations which a computer can execute for some domain-specific application.” This view brings along a tacit expectation to map (more or less) directly from a linguistic surface form to an actionable encoding of its intended meaning, e.g. in a database query or even programming language. In this tutorial, we embrace a broader perspective on semantic parsing as it has come to be viewed commonly in recent years. We will review graph-based meaning representations that aim to be applicationand domain-independent, i.e. seek to provide a reusable intermediate layer of interpretation that captures, in suitably abstract form, relevant constraints that the linguistic signal imposes on interpretation. Tutorial slides and additional materials are available at the following address: https://github.com/cfmrp/tutorial 2 Semantic Graph Banks In the first part of the tutorial, we will give a systematic overview of the available semantic graph banks. On the one hand, we will distinguish graph banks with respect to the facets of natural language meaning they aim to represent. For instance, some graph banks focus on predicate–argument structure, perhaps with some extensions for polarity or tense, whereas others capture (some) scopal phenomena. Furthermore, while the graphs in most graph banks do not have a precisely defined model theory in the sense of classical linguistic semantics, there are still underlying intuitions about what the nodes of the graphs mean (individual entities and eventualities in the world vs. more abstract objects to which statements about scope and presupposition can attach). We will discuss the different intuitions that underly different graph banks. On the other hand, we will follow Kuhlmann and Oepen (2016) in classifying graph banks with respect to the relationship they assume between the tokens of the sentence and the nodes of the graph (called anchoring of graph fragments onto input sub-strings). We will distinguish three flavors of semantic graphs, which by degree of anchoring we will call type (0) to type (2). While we use ‘flavor’ to refer to formally defined sub-classes of semantic graphs, we will reserve the term ‘framework’ for a specific linguistic approach to graph-based meaning representation (typically cast in a particular graph flavor, of course). Type (0) The strongest form of anchoring is obtained in bi-lexical dependency graphs, where graph nodes injectively correspond to surface lexical units (tokens). In such graphs, each node is directly linked to a specific token (conversely, there may be semantically empty tokens), and the nodes inherit the linear order of their corresponding tokens. This flavor of semantic graphs was popularized in part through a series of Semantic Dependency Parsing (SDP) tasks at the SemEval exercises in 2014–16 (Oepen et al., 2014, 2015; Che et al., 2016). Prominent linguistic frameworks instantiating this graph flavor include CCG word–word dependencies (CCD; Hockenmaier and Steedman, 2007), Enju Predicate– Argument Structures (PAS; Miyao and Tsujii, 2008), DELPH-IN MRS Bi-Lexical Dependencies (DM; Ivanova et al., 2012) and Prague Semantic Dependencies (PSD; a simplification of the tectogrammatical structures of Hajič et al., 2012). Type (1) A more general form of anchored semantic graphs is characterized by relaxing the correspondence relations between nodes and tokens, while still explicitly annotating the correspondence between nodes and parts of the sentence. Some graph banks of this flavor align nodes with arbitrary parts of the sentence, including subtoken or multi-token sequences, which affords more flexibility in the representation of meaning contributed by, for example, (derivational) affixes or phrasal constructions. Some further allow multiple nodes to correspond to overlapping spans, enabling lexical decomposition (e.g. of causatives or comparatives). Frameworks instantiating this flavor of semantic graphs include Universal Conceptual Cognitive Annotation (UCCA; Abend and Rappoport, 2013; featured in a SemEval 2019 task) and two variants of ‘reducing’ the underspecified logical forms of Flickinger (2000) and Copestake et al. (2005) into directed graphs, viz. Elementary Dependency Structures (EDS; Oepen and Lønning, 2006) and Dependency Minimal Recursion Semantics (DMRS; Copestake, 2009). All three frameworks serve as target representations in recent parsing research (e.g. Buys and Blunsom, 2017; Chen et al., 2018; Hershcovich et al., 2018). Type (2) Finally, our framework review will include Abstract Meaning Representation (AMR; Banarescu et al., 2013), which in our hierarchy of graph flavors is considered unanchored, in that the correspondence between nodes and tokens is not explicitly annotated. The AMR framework deliberately backgrounds notions of compositionality and derivation. At the same time, AMR frequently invokes lexical decomposition and represents some implicitly expressed elements of meaning, such that AMR graphs quite generally appear to ‘abstract’ furthest from the surface signal. Since the first general release of an AMR graph bank in 2014, the framework has provided a popular target for semantic parsing and has been the subject of two consecutive tasks at SemEval 2016 and 2017 (May, 2016; May and Priyadarshi, 2017). 3 Processing Semantic Graphs The creation of large-scale, high-quality semantic graph banks has driven research on semantic parsing, where a system is trained to map from natural-language sentences to graphs. There is now a dizzying array of different semantic parsing algorithms, and it is a challenge to keep track of their respective strengths and weaknesses. Different parsing approaches are, of course, more or less effective for graph banks of different flavors (and, at times, even specific frameworks). We will discuss these interactions in the tutorial and categorize existing approaches into four classes. Factorization-based approach A factorizationbased parser explicitly models the target semantic structures by defining a score function that is able to evaluate the “goodness” of any candidate graph. To make a score function computable, a parser usually factorizes the score of a graph into parts for smaller substrings and can then apply dynamic programming to search for the best graph. Composition-based approach Following the Principle of Compositionality, a semantic graph can be viewed as the result of a derivation process, in which a set of lexical and syntacticosemantic rules are iteratively applied and evaluated. A composition-based parser explicitly models such derivation structures by defining a symbolic system to manipulate graph construction and a score function to select preferable derivations. Transition-based approach A transition-based parser models a derivation process in a left-toright, word-by-word way. The key to building a high-accuracy parser is to define a score function that evaluates the individual derivation decisions for each token. In order to find a good derivation among a large set, a parser usually adopts a greedy search strategy which is sometimes psycholinguistically motivated. Translation-based approa