We investigate whether synthetic question-answer (QA) data generated by large language models (LLMs) can serve as an effective proxy for human-labeled benchmarks when the latter is unavailable. We assess the reliability of synthetic benchmarks across two experiments: one varying retriever parameters while keeping the generator fixed, and another varying the generator with fixed retriever parameters. Across four datasets, of which two open-domain and two proprietary, we find that synthetic benchmarks reliably rank the RAGs varying in terms of retriever configuration, aligning well with human-labeled benchmark baselines. However, they do not consistently produce reliable RAG rankings when comparing generator architectures. The breakdown possibly arises from a combination of task mismatch between the synthetic and human benchmarks, and stylistic bias favoring certain generators.

In Dutch parliamentary debates, over 95

Large language models (LLMs) are known to exhibit positional bias, the tendency of models to perform differently based on where relevant information appears within the input context. Understanding this bias is important in a retrieval augmented generation (RAG) setting, as it impacts how retrieved passages are taken into account by the model. We systematically investigate positional bias in a RAG setting, by evaluating four LLMs using three different types of distractor documents, assessing their ability to extract relevant information from the input context. Our findings reveal significant positional bias depending on the type of context documents used and the total amount of documents in the context. Furthermore, the results show that positional bias in state-of-the-art LLMs is not limited to information located in the middle of the input context. By analyzing the models’ attention, we identified patterns between the model’s accuracy in responding to questions and the correct attribution of attention to relevant information in the context. Our code is available at https://github.com/Janhutter/LBNOITM .

We study the WCAG compliancy and state of the metadata of PDF documents released under the Dutch Open Government Act (Woo). The results show that, in line with previous research on WCAG compliancy of PDF documents, only a fraction (0.2

We demo an IR system for municipal council meetings only available in video format. The system uses Whisper to turn speech into text. Three IR systems were compared in a user study: BM25/TF-IDF, KNN ranking on MPNet embeddings, and a chatbot prompted with the query and the top k highly ranked passages. Users solved complex information need slightly faster using the chatbot, but did not find the correct answer in 23

When Dutch citizens want to gain insights into the decision-making process of their government, they can file a so-called Freedom of Information Act request, requesting information on specific topics. The resulting documents (released publicly) have the potential to be a valuable resource for the research community, both in the domain of computer science, as well as the social- and political sciences. However, the current publication landscape is very scattered, with many organizations publishing on their own websites, with little to no coordination on document structure, (meta)data quality, and without a standardized metadata format. In this paper we present a collection of these documents published as FAIR data. The dataset contains just over two million pages, collected by scraping supplier websites, after which document metadata standardization was performed, and checks were carried out to ensure text- and metadata quality. The document text- and layout, their metadata, and where available links to the original PDF files, are all available through the DANS data repository, including usage instructions and examples.

The redaction of sensitive information in documents is common practice in specific types of organizations. This happens for example in court proceedings or in documents released under the Freedom of Information Act (FOIA). The ability to automatically detect when information has been redacted has several practical applications, such as the gathering of statistics on the amount of redaction present in documents, enabling a critical view on redaction practices. It can also be used to further investigate redactions, and whether or not the used techniques provide sufficient anonymization. The task is particularly challenging because of the large variety of redaction methods and techniques, from software for automatic redaction to manual redactions by pen. Any detection system must be robust to a large variety of inputs, as it will be run on many documents that might not even contain redactions. In this study, we evaluate two neural methods for the task, namely a Mask R-CNN model and a Mask2Former model, and compare them to a rule-based model based on optical character recognition and morphological operations. The best performing, the Mask R-CNN model, has a recall of .94 with a precision of .96 over a challenging data set containing several redaction types. Adding many pages without redaction barely lowers this score (precision drops to .90, recall drops to .92). The Mask2Former model is most robust to inputs without redactions, producing the least false positives of all models.

BCubed is a mathematically clean, elegant and intuitively well behaved external performance metric for clustering tasks. BCubed compares a predicted clustering to a known ground truth through elementwise precision and recall scores. For each element, the predicted and ground truth clusters containing the element are compared, and the mean over all elements is taken. We argue that BCubed overestimates performance, for the intuitive reason that the clustering gets credit for putting an element in its own cluster. This is repaired, and we investigate the repaired version, called "Elements Like Me (ELM)". We extensively evaluate ELM and conclude that it retains all positive properties of BCubed and gives a minimum 0 zero score when it should.

This paper presents an effective and efficient approach for automatic extraction of key features from enforcement decisions, such as their legal basis and their legal effect, by strategically applying a Large Language Model (LLM) on top of rule-based methods. Initially, rule-based methods identify candidate sentences within these decisions containing these features, after which these sentences are analyzed by GPT-3.5 to extract the features. This approach is efficient as it reduces the input and number of resources needed for effective and context aware information extraction. Furthermore, other features that have not been subject to a rule-based selection first can be extracted by an LLM from the same set of candidate sentences when they exist in close proximity of each other.

The Panoptic Quality metric, developed by Kirillov et al. in 2019, makes object-level precision, recall and F1 measures available for evaluating image segmentation, and more generally any partitioning task, against a gold standard. Panoptic Quality is based on partial isomorphisms between hypothesized and true segmentations. Kirillov et al. desire that functions defining these one-to-one matchings should be simple, interpretable and effectively computable. They show that for t and h, true and hypothesized segments, the condition stating that there are more correct than wrongly predicted pixels, formalized as IoU(t,h)>.5 or equivalently as |t∩h|>.5|t∪h| has these properties. We show that a weaker function, requiring that more than half of the pixels in the hypothesized segment are in the true segment and vice-versa, formalized as |t∩h|>.5|t| and |t∩h|>.5|h|, is not only sufficient but also necessary. With a small proviso, every function defining a partial isomorphism satisfies this condition. We theoretically and empirically compare the two conditions.

In recent years, an increasing number of companies and institutions have begun the process of digitizing their physical records to promote digital access and searchability of their collections. For cost-efficiency, documents are often scanned in consecutively, resulting in large PDF files consisting of many documents. Although cost-effective, this practice can be harmful for searchability when these concatenated documents are used to build a search engine. The task of Page Stream Segmentation is concerned with recovering the original document boundaries through the analysis of the text and/or images of these PDF files. Currently, many of the approaches to solving this problem make use of machine learning techniques that require significant amounts of training data. However, due to the sometimes sensitive nature of the data, few large datasets exist, and there is a lack of agreed-upon metrics to measure system performance. In an effort to resolve these issues and provide a comprehensive overview of the state of the field, we constructed the OpenPSS benchmark, consisting of two large public datasets and a comprehensive study of various types of approaches, evaluated using multiple evaluation metrics. The datasets originated from several Dutch government institutions, cover a heterogeneous set of topics, and total roughly 141 thousand pages from around 32 thousand documents. The experimental results show that ensemble methods using both the text and image representations of pages are superior to uni-modal methods, and that image-based neural methods are not as robust as text models when evaluated on out-of-distribution data.

In the Netherlands, the Open Government Act (Wet openbare overheid or Woo/Wob in Dutch) is in effect, with the primary objective of ensuring a more transparent government. In line with the legislation, a search engine named Woogle has been designed and developed to centralize documents published under the Open Government Act. The Estonian Public Information Act serves a similar purpose and requires all public institutions to publish information generated during official duties, fostering transparency and public oversight. Currently, Estonia’s document repositories are decentralized, and content search is not supported, which hinders people’s ability to efficiently locate information. This study aims to assess public information accessibility in Estonia and to apply Woogle’s design and techniques to Estonia’s document repositories, thereby evaluating its potential for broader European implementation. The methodology involved web scraping data and documents from 57 Estonian public institutions’ document repositories. The results indicate that Woogle’s design and techniques can be implemented in Estonia. From a technical perspective, the alignment of the fields was successful, while it was found that content-wise, the Estonian data present challenges due to inconsistencies and lack of comprehensive categorization. The findings suggest potential scalability across European countries, pointing to a broader applicability of the Woogle model for creating a corpus of Freedom of Information Act documents in Europe. The collected data are available as a dataset.

This article investigates under which video watch conditions YouTube's recommender system tends to develop a preference for conspiracy-classified videos. Whereas existing research on so-called filter bubbles and rabbit holes tends to rely on non-personalized recommendations and on standard watch patterns, this study puts personalization and diversified user strategies at the center of its design. 20 authenticated bots have been instructed to watch YouTube content based on four distinct watch strategies. In a baseline strategy, bots watched non-conspiracy videos only. Treatment strategies involved watching conspiracy-classified content, selected based on either non-personalized, partly-personalized, or fully-personalized input. Bots watched a total of 15 videos, and after each video their top 20 homepage recommendations were collected and classified as either conspiracy-related or not. This allowed us to measure the impact of each video watched and of each watch strategy on the proportion of conspiracy-classified content recommended at each step. The same experiment has been reverted, exposing the treatment groups to non-conspiracy videos only, to assess the persistence of this pattern. Our results show that users primed with conspiracy-classified content tend to quickly receive a much larger proportion of conspiracy-classified recommendations. Inverting this pattern proves significantly more difficult than generating it. There are also indications that watch strategies relying on personalized content as input might produce stronger effects. This article contributes evidence to the argument that YouTube's recommendation system is prone to generating strong, potentially pernicious recommendation patterns. Moreover, it contributes a replicable methodology that puts personalization at the center of the stage in the study of content personalization algorithms.

The task of coreference resolution concerns the clustering of words and phrases referring to the same entity in text, either in the same document or across multiple documents. The task is challenging, as it concerns elements of named entity recognition and reading comprehension, as well as others. In this paper, we introduce DutchParliament, a new Dutch coreference resolution dataset obtained through the manual annotation of 74 government debates, expanded with a domain-specific class. In contrast to existing datasets, which are often composed of news articles, blogs or other documents, the debates in DutchParliament are transcriptions of speech, and therefore offer a unique structure and way of referencing compared to other datasets. By constructing and releasing this dataset, we hope to facilitate the research on coreference resolution in niche domains, with different characteristics than traditional datasets. The DutchParliament dataset was compared to SoNaR-1 and RiddleCoref, two other existing Dutch coreference resolution corpora, to highlight its particularities and differences from existing datasets. Furthermore, two coreference resolution models for Dutch, the rule-based DutchCoref model and the neural e2eDutch model, were evaluated on the DutchParliament dataset to examine their performance on the DutchParliament dataset. It was found that the characteristics of the DutchParliament dataset are quite different from that of the other two datasets, although the performance of the e2eDutch model does not seem to be significantly affected by this. Furthermore, experiments were conducted by utilizing the metadata present in the DutchParliament corpus to improve the performance of the e2eDutch model. The results indicate that the addition of available metadata about speakers has a beneficial effect on the performance of the model, although the addition of the gender of speakers seems to have a limited effect.

Page stream segmentation (PSS) is the task of retrieving the boundaries that separate source documents given a consecutive stream of documents (for example, sequentially scanned PDF files). The task has recently gained more interest as a result of the digitization efforts of various companies and organizations, as they move towards having all their documents available online for improved searchability and accessibility for users. The current state-of-the-art approach is neural start of document page classification on representations of the text and/or images of pages using models such as Visual Geometry Group-16 (VGG-16) and BERT to classify individual pages. We view the task of PSS as a clustering task instead, hypothesizing that pages from one document are similar to each other and different to pages in other documents, something that is difficult to incorporate in the current approaches. We compare the segmentation performance of an agglomerative clustering method with a binary classification model based on images on a new publicly available dataset and experiment with using either pretrained or finetuned image vectors as inputs to the model. To adapt the clustering method to PSS, we propose the switch method to alleviate the effects of pages of the same class having a high similarity, and report an improvement in the scores using this method. Unfortunately, neither clustering with pretrained embeddings nor clustering with finetuned embeddings outperformed start of document page classification for PSS. However, clustering with either pretrained or finetuned representations is substantially more effective than the baseline, with finetuned embeddings outperforming pretrained embeddings. Finally, having the number of documents K as part of the input, in our use case a realistic assumption, has a surprisingly significant positive effect. In contrast to earlier papers, we evaluate PSS with the overlap weighted partial match F1 score, developed as a Panoptic Quality in the computer vision domain, a metric that is particularly well-suited to PSS as it can be used to measure document segmentation.

Government transparency is central in a democratic society, and increasingly governments at all levels are required to publish records and data either proactively, or upon so-called Freedom of Information (FIA) requests. However, public bodies who are required by law to publish many of their documents turn out to have great difficulty to do so. And what they publish often is in a format that still breaches the requirements of the law, stipulating principles comparable to the FAIR data principles. Hence, this demo is addressing a timely problem: the FAIR publication of FIA dossiers, which is obligatory in The Netherlands since May 1st 2022.