Scientific evidence is predominantly communicated through unstructured articles, while many downstream workflows require structured representations of study populations, variables, methods, and quantitative findings. Large language models (LLMs) offer a promising route toward automating this transformation, yet their reliability in constructing schema-constrained evidence records from parsed scientific full texts remains insufficiently understood. Evidence synthesis is a representative use case in which systematic reviews and meta-analyses require accurate, structured study information before quantitative integration can begin. We present a schema-grounded, tiered evaluation framework for structured scientific evidence extraction that increases task difficulty from single-entity identification to relational tuples and corpus-level statistical tasks. We evaluate frontier commercial and open-weight models across five scientific domains under a stricter multi-level assessment protocol, complemented by human validation of model-based judging and process-level inspection of intermediate reasoning for statistical tasks. Results reveal a clear pattern: models perform substantially better on low-complexity extraction tasks than on tasks requiring correct bindings among entities, numerical values, and methodological attributes. Intermediate-step analysis further indicates error amplification under exact downstream aggregation, where small upstream errors often prevent correct final outputs. Scaling model size yields uneven gains and does not remove these weaknesses. These findings show that recent advances have not resolved the fundamental challenges of structured scientific evidence extraction. We provide a reusable query suite and diagnostic evaluation framework to support future fine-grained benchmarks and more rigorous assessment of models for evidence-intensive scientific workflows.
Large Language Models (LLMs) are increasingly used as flexible access layers for information-rich collections, raising the question of whether they can also support structured access to cultural heritage data as if they were digital libraries. This work investigates that question through Galois, a database-inspired system that treats the LLM as a noisy storage layer. We evaluate six querying modes: direct natural language prompting, direct SQL prompting, and four Galois strategies that combine LLM-facing tuple retrieval with deterministic relational processing. The evaluation uses two cultural heritage benchmarks: Paintings Extended, based on paintings data, and a new benchmark based on Europeana metadata. For both benchmarks, we define fourteen query templates with ten variants each. The main experiments use GPT-4o-mini, Granite 3.3 8B, and Llama 3 70B, with an additional diagnostic analysis of two GPT-5 deployments. We complement the benchmark metrics with a stratified manual audit of 60 unique disagreements. Performance is strongly shaped by model capability, deployment behavior, dataset structure, and execution strategy. Direct natural language prompting is the strongest and most robust baseline, while direct SQL prompting becomes nearly equivalent on the strongest open models. Among the structured strategies, only GaloisA and GaloisF become meaningfully competitive, especially on the flatter Europeana benchmark. In the audited sample, 35 disagreements were model-originated failures, 21 reflected reference-coverage or metadata-normalization effects, and four remained indeterminate. The additional GPT-5 deployment analysis reveals frequent empty outputs and provider-level constraints. These diagnostic results characterize the specific tested deployments and must not be interpreted as a general ranking of GPT-5 model capability. Querying LLMs as digital libraries is feasible, but its effectiveness depends on model strength, deployment conditions, dataset structure, and execution strategy. Direct prompting is already strong for exploratory cultural heritage access, while Galois remains valuable when relational discipline and controlled query execution are required.
At its core, a repository migration is the transfer of records from one system to another. In some cases, it is more of an unraveling, repair, relocation, and rebuilding of a historic property. In 2024, the International Food Policy Research Institute (IFPRI) library undertook its second migration from an independent CONTENTdm repository to a shared DSpace repository. The institute set an ambitious goal to complete the migration within a single contract cycle with CONTENTdm. Since 2012, the repository has functioned as a bibliographic metadata layer serving the main corporate website, numerous project sites, and the global consortium (CGIAR) website, as well as a digital publishing platform and institutional metrics and reporting system. The team started working with a combination of tools, including: Microsoft Excel, OpenRefine, and custom scripts. Microsoft Excel allowed staff to easily see and manipulate large amounts of data across different metadata fields. OpenRefine (version 3.7) was instrumental in normalizing and cleaning name and keyword variations, as well as locating and editing a single repeated datapoint. The scripts, created by a member of the team and author of this paper, saved hours of time and allowed the team to meet ambitious goals. The team successfully migrated over 20,000 records within 13 months while simultaneously improving metadata quality. Prior to the migration, about half of journal articles lacked external links and several dozen PDFs had been lost over time. Additionally, from 2008–2017, IFPRI’s records experienced an average of 40
Terminology and entity annotation tasks in digital-library and corpus-oriented workflows often involve expressions that are discontinuous, overlapping, or partially shared. While several existing annotation environments support span-based and relation-based annotation, discontinuous structures are frequently represented indirectly through workaround mechanisms, such as linking independent spans via auxiliary relations. This may increase annotation complexity and reduce the transparency of the resulting annotation objects. This paper introduces Annotaterm, a lightweight web-based annotation environment designed to support the direct annotation of contiguous, discontinuous, and overlapping terms through a fragment-based interaction model. This model supports simple, composite, and overlapping annotations in a uniform and inspectable way. A preliminary user evaluation with 14 participants suggests that the tool is perceived as suitable for terminology-oriented annotation tasks, particularly for handling overlapping and discontinuous expressions, while also identifying areas for future refinement. The proposed approach contributes both a conceptual model and a practical annotation environment for digital-library and terminology-oriented workflows requiring transparent handling of complex textual structures.
Conversational search systems enable natural, coherent dialogues between users and search systems to satisfy the user information need. The best of such systems improve user experience by asking clarifying questions (CQs) to resolve ambiguity in user queries. However, generating CQs that are both relevant and conversationally appropriate remains a significant challenge. To this end, we use Google’s “People Also Ask” (PAA) feature, grounding our questions in real-life user search behavior. We identify the most suitable methodology for this task by conducting an extensive comparative analysis of two methods: Parameter-Efficient Fine-Tuning (PEFT) and prompt engineering, and evaluate these using three state-of-the-art, lightweight language models. Furthermore, we introduce two new, human-annotated test sets derived from various conversational search datasets. Our results demonstrate that PEFT consistently and significantly outperforms all prompt engineering approaches across all models and test sets, and is statistically comparable to traditional full fine-tuning. These findings suggest that for the PAA-to-CQ task, investing in a high-quality dataset for efficient fine-tuning is a more reliable path to achieving high-quality, stylistically consistent outputs than relying on in-context learning alone.
