Multilingual question answering (QA) over linked data can make structured knowledge accessible across languages. However, most state-of-the-art QA systems are limited to English and rely on machine learning methods that lack controllability and extensibility, making their governance and incremental improvement challenging, along with the effort needed to collect and provide training data. To address this, we follow a model-based approach to QA that builds on a declarative model of the lexicon-ontology interface, OntoLex-Lemon, which enables the specification of the meaning of lexical entries with respect to the vocabulary of a particular dataset. From such a lexicon, in our approach, a QA grammar is automatically generated that can be used to parse questions into SPARQL queries. We demonstrate the extensibility of the approach to different languages by adapting it to German, Italian, and Spanish. We evaluate the approach with respect to the QALD benchmarks and show that our approach outperforms the state-of-the-art on all these datasets in an incremental evaluation mode in which additional lexical entries for test data are added. For example, on QALD-9, our approach obtains scores of (English), (German), (Italian), and (Spanish) in an incremental evaluation mode in which lexical entries covering test data questions are added. So far there is no system described in the literature that works for at least four languages while reaching state-of-the-art performance on all of them. Finally, we demonstrate the low efforts necessary to port the system to a new dataset and vocabulary.

Transition Metal Complexes (TMCs) have wide-ranging practical utility in chemistry, with possible applications that range from catalysis to medicinal chemistry. The study of TMCs and their properties is thus a field rich with potential, one in which machine learning and computational approaches can offer a substantial aid. For this reason, appropriate and accessible datasets, collecting a wide range of information, are required in order to facilitate the effective analysis and investigation of such compounds. This paper contributes to the data modelling effort via the introduction of the transition metal quantum mechanics RDF (tmQM-RDF) dataset, a knowledge graph constructed using the Resource Description Framework (RDF) vocabulary which collects rich and detailed descriptions of approximately 50k TMCs. These descriptions are both qualitative and quantitative in nature, encompassing the compositional nature of TMCs in terms of their constituting ligands, as well as the entirety of their molecular graphs. An example of the power of the proposed representation is presented, showcasing how the information available in tmQM-RDF can be exploited for TMC manipulation tasks, achieving promising performance even with relatively simple probabilistic models.

Refrigeration based on the magnetocaloric effect (MCE) can contribute to energy-saving, environmentally friendly cooling in private households, or industrial application. The cooling is based on the reversible heat release or uptake during a phase-transformation of the materials that can be controlled by a magnetic field. This process could replace conventional compression-based refrigeration, which often relies on environmentally harmful refrigerants. The MCE is observed in a large number of magnetic alloys, upon them are Heusler alloys. Thus, optimization of MCE materials involves the screening and testing of many different materials systems, generating a large amount of data that, so far, has not been organized in a systematic manner, hindering the progress of the field. Here, were present an approach to digitalize the process chain from synthesis, experiment, and simulation to prototypical applications. Different Heusler alloys have been examined experimentally as model systems for potential applications in magnetic cooling. Templates based on the OTTR technology have been developed and implemented for the acquisition and semantic representation of knowledge in the development of an ontology. The ontology, when combined with unstructured data, can be exploited to train a model that can then be used to predict missing facts, which can help to gain new insights and to generate new hypotheses. Furthermore, tools have been developed that automate and accelerate data acquisition into ontological structures, and workflows have been implemented that provide a fast, easy-to-use theoretical and experimental evaluation of the MCE from first principles and raw data.

Explainability is widely regarded as essential for trustworthy artificial intelligence systems. However, the metrics commonly used to evaluate counterfactual explanations are algorithmic evaluation metrics that are rarely validated against human judgments of explanation quality. This raises the question of whether such metrics meaningfully reflect user perceptions. We address this question through an empirical study that directly compares algorithmic evaluation metrics with human judgments across three datasets. Participants rated counterfactual explanations along multiple dimensions of perceived quality, which we relate to a comprehensive set of standard counterfactual metrics. We analyze both individual relationships and the extent to which combinations of metrics can predict human assessments. Our results show that correlations between algorithmic metrics and human ratings are generally weak and strongly dataset-dependent. Moreover, increasing the number of metrics used in predictive models does not lead to reliable improvements, indicating structural limitations in how current metrics capture criteria relevant for humans. Overall, our findings suggest that widely used counterfactual evaluation metrics fail to reflect key aspects of explanation quality as perceived by users, underscoring the need for more human-centered approaches to evaluating explainable artificial intelligence.

Link Prediction (LP) approaches based on Language Models (LMs) operate over the labels and descriptions of entities and relations in a Knowledge Graph (KG). Recent approaches have shown that incorporating a local graph neighborhood can improve the LP capabilities of LMs. These approaches usually sample a context from the neighborhood around a query triple randomly, thereby incorporating noise that might hinder the model in making correct predictions. In this work, we derive an approximately optimal context for a given query under the assumption that we know the correct answer. This allows us to investigate the characteristics of such contexts, thereby providing an approximate upper bound on the achievable performance when using optimal contexts.

We introduce graph pattern-based association rules (GPARs) for directed labeled multigraphs such as RDF graphs. GPARs support both generative tasks, where a graph is extended, and evaluative tasks, where the plausibility of a graph is assessed. The framework goes beyond related formalisms such as graph functional dependencies, graph entity dependencies, relational association rules, graph association rules, multi-relation and path association rules, and Horn rules. Given a collection of graphs, we evaluate graph patterns under no-repeated-anything semantics, which allows the topology of a graph to be taken into account more effectively. We define a probability space and derive confidence, lift, leverage, and conviction in a probabilistic setting. We further analyze how these metrics relate to their classical itemset-based counterparts and identify conditions under which their characteristic properties are preserved.

Knowledge Graph Question Answering (KGQA) aims to generate factual answers to natural language (NL) questions by querying structured data in knowledge graphs (KGs). This study investigates the use of in-context learning (ICL) for text-to-SPARQL generation, focusing on how the number, similarity, and diversity of prompt examples can affect an LLM' performance when applied to the text-to-SPARQL task. We conducted experiments using the LC-QuAD 1.0 benchmark and a 70B-parameter instruction-tuned LLM, exploring multiple dynamic example selection strategies based on BERT embeddings. We evaluated performance based on the retrieved answers and the structural similarity of the queries. We found that adding similar examples to the prompt enhanced performance; however, the gains became minimal beyond approximately ten examples. Contrary to our initial hypothesis, diversity did not yield the expected improvements. We also conducted an error analysis and further experiments, revealing a recurring issue: even when gold URIs are provided, the models often generate fake URIs, and additional problems persist.

This article describes advancements in the ongoing digital transformation in materials science and engineering. It is driven by domain-specific successes and the development of specialized digital data spaces. There is an evident and increasing need for standardization across various subdomains to support science data exchange across entities. The MaterialDigital Initiative, funded by the German Federal Ministry of Education and Research, takes on a key role in this context, fostering collaborative efforts to establish a unified materials data space. The implementation of digital workflows and Semantic Web technologies, such as ontologies and knowledge graphs, facilitates the semantic integration of heterogeneous data and tools at multiple scales. Central to this effort is the prototyping of a knowledge graph that employs application ontologies tailored to specific data domains, thereby enhancing semantic interoperability. The collaborative approach of the Initiative's community provides significant support infrastructure for understanding and implementing standardized data structures, enhancing the efficiency of data-driven processes in materials development and discovery. Insights and methodologies developed via the MaterialDigital Initiative emphasize the transformative potential of ontology-based approaches in materials science, paving the way toward simplified integration into a unified, consolidated data space of high value.

In recent years, large language models have achieved state-of-the-art performance across various NLP tasks. However, investigations have shown that these models tend to rely on shortcut features, leading to inaccurate predictions and causing the models to be unreliable at generalization to out-of-distribution (OOD) samples. For instance, in the context of relation extraction (RE), we would expect a model to identify the same relation independently of the entities involved in it. For example, consider the sentence "Leonardo da Vinci painted the Mona Lisa" expressing the created(Leonardo_da_Vinci, Mona_Lisa) relation. If we substiute "Leonardo da Vinci" with "Barack Obama", then the sentence still expresses the created relation. A robust model is supposed to detect the same relation in both cases. In this work, we describe several semantically-motivated strategies to generate adversarial examples by replacing entity mentions and investigate how state-of-the-art RE models perform under pressure. Our analyses show that the performance of these models significantly deteriorates on the modified datasets (avg. of -48.5 extent on shortcuts, such as surface forms (or patterns therein) of entities, without making full use of the information present in the sentences.

Download This Paper Open PDF in Browser Add Paper to My Library Share: Permalink Using these links will ensure access to this page indefinitely Copy URL Copy DOI

Dataset search, or more specifically, ad hoc dataset retrieval which is a trending specialized IR task, has received increasing attention in both academia and industry. While methods and systems continue evolving, existing test collections for this task exhibit shortcomings, particularly suffering from lexical bias in pooling and limited to keyword-style queries for evaluation. To address these limitations, in this paper, we construct ACORDAR 2.0, a new test collection for this task which is also the largest to date. To reduce lexical bias in pooling, we adapt dense retrieval models to large structured data, using them to find an extended set of semantically relevant datasets to be annotated. To diversify query forms, we employ a large language model to rewrite keyword queries into high-quality question-style queries. We use the test collection to evaluate popular sparse and dense retrieval models to establish a baseline for future studies. The test collection and source code are publicly available.

Link Prediction(LP) is an essential task over Knowledge Graphs(KGs), traditionally focussed on using and predicting the relations between entities. Textual entity descriptions have already been shown to be valuable, but models that incorporate numerical literals have shown minor improvements on existing benchmark datasets. It is unclear whether a model is actually better in using numerical literals, or better capable of utilizing the graph structure. This raises doubts about the effectiveness of these methods and about the suitability of the existing benchmark datasets. We propose a methodology to evaluate LP models that incorporate numerical literals. We propose i) a new synthetic dataset to better understand how well these models use numerical literals and ii) dataset ablations strategies to investigate potential difficulties with the existing datasets. We identify a prevalent trend: many models underutilize literal information and potentially rely on additional parameters for performance gains. Our investigation highlights the need for more extensive evaluations when releasing new models and datasets.

Within industry, it is vital to adequately commu-nicate the qualities and features of what is to be built, and requirements are important artefacts for this purpose. Having machine-readable re-quirements can enhance the level of control over the requirements, allowing more efficient requirement management and communication. Training a semantic parser typically requires a dataset with thousands of examples. However, creating such a dataset for textual require-ments poses significant challenges. In this study, we investigate to what extent a large language model can assist a human annotator in creating a gold corpus for semantic parsing of textual requirements. The language model generates a semantic parse of a textual requirement that is then corrected by a human and then added to the gold standard. Instead of incrementally fine-tuning the language model on the growing gold standard, we investigate different strategies of including examples from the growing gold standard in the prompt for the language model. We found that selecting the requirements most semantically similar to the target sentence and ordering them with the most similar requirement first yielded the best performance on all the metrics we used. The approach resulted in 41 % fewer edits compared to creating the parses from scratch, – thus, significantly less human effort is involved in the creation of the gold standard in collaborative annotation. Our findings indicate that having more requirements in the gold standard improves the accuracy of the initial parses.

Industry requirements describe the qualities that a project or a service must provide.Most requirements are, however, only available in natural language format and are embedded in textual documents.To be machineunderstandable, a requirement needs to be represented in a logical format.We consider that a requirement consists of a scope, which is the requirement's subject matter, a condition, which is any condition that must be fulfilled for the requirement to be relevant, and a demand, which is what is required.We introduce a novel task, the identification of the semantic components scope, condition, and demand in a requirement sentence, and establish baselines using sequence labelling and few-shot learning.One major challenge with this task is the implicit nature of the scope, often not stated in the sentence.By including document context information, we improved the average performance for scope detection.Our study provides insights into the difficulty of machine understanding of industry requirements and suggests strategies for addressing this challenge.

An open issue for Semantic Question Answering Systems is bridging the so called lexical gap , referring to the fact that the vocabulary used by users in framing a question needs to be interpreted with respect to the logical vocabulary used in the data model of a given knowledge base or knowledge graph. Building on previous work to automatically induce ontology lexica from language corpora by using association rules to identify correspondences between lexical elements on the one hand and ontological vocabulary elements on the other, in this paper we propose LexExMachinaQA, a framework allowing us to evaluate the impact of automatically induced lexicalizations in terms of alleviating the lexical gap in QA systems. Our framework combines the LexEx-Machina approach (Ell et al., 2021) for lexicon induction with the QueGG system proposed by Benz et al. (Benz et al., 2020) that relies on grammars automatically generated from ontology lexica to parse questions into SPARQL. We show that automatically induced lexica yield a decent performance i.t.o. F 1 measure with respect to the QLAD-7 dataset, representing a 34% – 56% performance degradation with respect to a manually created lexicon. While these results show that the fully automatic creation of lexica for QA systems is not yet feasible, the method could certainly be used to bootstrap the creation of a lexicon in a semi-automatic manner, thus having the potential to significantly reduce the human effort involved.

OTTR is a language for representing ontology modeling patterns, which enables to build ontologies or knowledge bases by instantiating templates. Thereby, particularities of the ontological representation language are hidden from the domain experts, and it enables ontology engineers to, to some extent, separate the processes of deciding about what information to model from deciding about how to model the information, e.g., which design patterns to use. Certain decisions can thus be postponed for the benefit of focusing on one of these processes. To date, only few works on ontology engineering where ontology templates are applied are described in the literature. In this paper, we outline our methodology and report findings from our ontology engineering activities in the domain of Material Science. In these activities, OTTR templates play a key role. Our ontology engineering process is bottom-up, as we begin modeling activities from existing data that is then, via templates, fed into a knowledge graph, and it is top-down, as we first focus on which data to model and postpone the decision of how to model the data. We find, among other things, that OTTR templates are especially useful as a means of communication with domain experts. Furthermore, we find that because OTTR templates encapsulate modeling decisions, the engineering process becomes flexible, meaning that design decisions can be changed at little cost.

Background Evidence-based medicine propagates that medical/clinical decisions are made by taking into account high-quality evidence, most notably in the form of randomized clinical trials. Evidence-based decision-making requires aggregating the evidence available in multiple trials to reach -by means of systematic reviews- a conclusive recommendation on which treatment is best suited for a given patient population. However, it is challenging to produce systematic reviews to keep up with the ever-growing number of published clinical trials. Therefore, new computational approaches are necessary to support the creation of systematic reviews that include the most up-to-date evidence.We propose a method to synthesize the evidence available in clinical trials in an ad-hoc and on-demand manner by automatically arranging such evidence in the form of a hierarchical argument that recommends a therapy as being superior to some other therapy along a number of key dimensions corresponding to the clinical endpoints of interest. The method has also been implemented as a web tool that allows users to explore the effects of excluding different points of evidence, and indicating relative preferences on the endpoints. Results Through two use cases, our method was shown to be able to generate conclusions similar to the ones of published systematic reviews. To evaluate our method implemented as a web tool, we carried out a survey and usability analysis with medical professionals. The results show that the tool was perceived as being valuable, acknowledging its potential to inform clinical decision-making and to complement the information from existing medical guidelines. Conclusions The method presented is a simple but yet effective argumentation-based method that contributes to support the synthesis of clinical trial evidence. A current limitation of the method is that it relies on a manually populated knowledge base. This problem could be alleviated by deploying natural language processing methods to extract the relevant information from publications.

Knowledge Graphs are relevant for many applications, but are inherently incomplete. Thus, Link Prediction methods have been proposed to infer new triples in order to complete a given Knowledge Graph. Many Link Prediction methods ignore literals, in spite of the fact that literals can express important information about entities not encoded in relations between entities. The existing methods that do incorporate literal information (e. g., LiteralE) introduce complex architectures by modifying the model or the loss-function. In our research paper, we propose a new approach that relies on graph transformations to transform a graph in such a way that existing Link Prediction methods can leverage the literal information. In particular, we define three transformations and evaluate them in comparison to state-of-the-art approaches. In most cases, the additional triples generated by our transformations lead to a performance increase and even state-of-the-art performance can be reached when comparing against LiteralE. It turned out that even a reductionistic transformation is able to archive comparable results like current, more complex, state-of-the-art approaches which incorporate literals.