Evaluating Knowledge Graph Question Answering (KGQA) systems is increasingly unreliable due to static benchmarks, evolving Knowledge Graphs (KG), and memorization effects in Large Language Models (LLM). We present DynBench Generator, a web frontend for automatically generating new KGQA benchmark datasets that both preserve the original question and query complexity and are resistant to LLM memorization. In this demo, we showcase an interactive platform that enables on-demand generation and inspection of memorization-resistant question-query pair for fair and reproducible KGQA evaluation.

Knowledge Graph Question Answering (KGQA) systems transform natural-language questions into structured SPARQL queries to retrieve information from Knowledge Graphs (KGs). However, existing KGQA benchmarks are static, prone to obsolescence as KGs evolve, and increasingly unreliable for evaluating systems based on Large Language Models (LLMs) due to memorization effects (i.e., a model has reproduced the correct output from existing training data). This paper presents DynBench, a fully automated and robust framework for the dynamic generation of KGQA benchmark datasets. Unlike prior approaches, DynBench is enabled to automatically generate an arbitrary number of new benchmark datasets while preserving the natural-language surface forms and structural complexity of the original SPARQL queries in comparison to the given original KGQA dataset. Using entity and property substitutions within the same KG, DynBench produces semantically consistent question-query pairs and automatically validates them without human intervention. We introduce an automatic validation mechanism that ensures semantic and syntactic integrity through back-transformation and metric-based validation. Two dynamic bench-marks were generated from the well-known datasets QALD-9-plus as well as LC-QuAD 2.0 and evaluated using both human experts and automated metrics to show DynBench’s dataset-agnostic capabilities. Given our results, we can infer that the Levenshtein distance serves as the most reliable automatic validation measure, achieving a precision of up to 0.96 and demonstrating a strong correlation with human assessments. DynBench thus enables scalable, repeatable, and memorization-resistant dataset generation—providing a foundation for sustainable and fair evaluation of LLM-based KGQA systems.

As diverse linguistic backgrounds contribute valuable insights to scientific research, effective Cross-Lingual Semantic Search (CLSS) mechanisms, which often remain overlooked, become crucial. This paper introduces CLASS MATE(https://lass-kg.demos.dice-research.org/)-a CLSS application working over material science knowledge graphs (KGs). Our work aims to bridge the digital language divide in the research community by employing advanced knowledge representation techniques. In particular, (1) we acquire our KG containing chemical substances with multilingual entity labels; (2) we implement a symbolic similarity-based named entity recognition algorithm; and (3) we develop a demo application employing the previous steps for retrieving information requested by a user from our KG and LOD sources in multiple languages. Our industry partner Springer Nature provided us with a KG as an information source to understand information needs. To the best of our knowledge, we made the first contribution to CLSS within material science.

Over time, software systems have reached a level of complexity that makes it difficult for their developers and users to explain particular decisions made by them. In this paper, we focus on the explainability of component-based systems for Question Answering (QA). These components often conduct processes driven by AI methods, in which behavior and decisions cannot be clearly explained or justified, s.t., even for QA experts interpreting the executed process and its results is hard. To address this challenge, we present an approach that considers the components' input and output data flows as a source for representing the behavior and provide explanations for the components, enabling users to comprehend what happened. In the QA framework used here, the data flows of the components are represented as SPARQL queries (inputs) and RDF triples (outputs). Hence, we are also providing valuable insights on verbalization regarding these data types. In our experiments, the approach generates explanations while following template-based settings (baseline) or via the use of Large Language Models (LLMs) with different configurations (automatic generation). Our evaluation shows that the explanations generated via LLMs achieve high quality and mostly outperform template-based approaches according to the users' ratings. Therefore, it enables us to automatically explain the behavior and decisions of QA components to humans while using RDF and SPARQL as a context for explanations.

This paper presents insights into a real-world railway ontology and a corresponding Knowledge Graph Question Answering (KGQA) approach within the domain of train and track operations, developed in collaboration with DB Systel GmbH (subsidiary of the main German railway company). Our solution enables seamless access to infrastructure elements, train schedules, and operational data via a question-answering system powered by Large Language Models (LLMs). To ensure high quality, cost efficiency, and runtime performance, LLMs are utilized exclusively for SPARQL query generation, striking a balance between flexibility and control. This approach significantly enhances the precision and efficiency of information retrieval, making railway data more accessible and actionable. We discuss the implementation, evaluation results, and key trade-offs of our approach.

Large Language Models (LLMs) demonstrate impressive capabilities in Question Answering tasks, yet previous research suggests current approaches fail to accurately reflect their real SPARQL query generation capabilities due to memorization effects caused by integrating benchmark datasets into the training data. These effects artificially inflate perceived performance, i.e., the quality of the showing by LLMs cannot actually be achieved on previously unseen queries. This paper presents DynBench – a novel approach to creating high-quality benchmarking datasets that address this challenge in the field of Knowledge Graph Question Answering (KGQA). We develop new datasets based on two datasets – QALD-9-Plus and LC-QuAD – by systematically replacing entities in SPARQL queries with alternatives retrieved from Wikidata and within the corresponding natural-language questions. Our findings confirm that the proposed approach successfully creates new benchmark datasets that can be used for evaluating KGQA systems. Therefore, our approach drastically reduces the risk of memorization effects, thereby increasing the trustworthiness of benchmark results for LLM-based KGQA approaches.

Nowadays, the importance of software with natural-language user interfaces cannot be underestimated. In particular, in Question Answering (QA) systems, generating a SPARQL query for a given natural-language question (often named Query Building) from the information retrieved from the same question is the central task of QA systems working over Knowledge Graphs (KGQA). Due to the rise of Large Language Models (LLMs), they are considered a well-suited method to increase the quality of the question-answering functionality, as there is still a lot of room for improvement, aiming for enhanced quality and trustworthiness. However, LLMs are trained on web data, where researchers have no control over whether the benchmark or the knowledge graph was already included in the training data. In this paper, we introduce a novel method that evaluates the quality of LLMs by generating a SPARQL query from a natural-language question under various conditions: (1) zero-shot SPARQL generation, (2) with knowledge injection, and (3) with "anonymized" knowledge injection. This enables us, for the first time, to estimate the influence of the training data on the QA quality improved by LLMs. Ultimately, this will help to identify how portable a method is or whether good results might mostly be achieved because a benchmark was already included in the training data (cf. LLM memorization). The developed method is portable, robust, and supports any knowledge graph; therefore, it could be easily applied to any KGQA or LLM, s.t., generating consistent insights into the actual LLM capabilities is possible.

This article introduces foundational components for decentralized B2B data sharing based on the solid protocol, emphasizing data sovereignty, security, and interoperability. These components are: (1) Authorization app (AuthApp) – facilitating granular control and compliance in access granting and revocation processes; (2) rights delegation proxy (RDP) – supporting controlled delegation of rights, enabling natural persons to act on behalf of organizations while ensuring privacy and traceability; (3) data provisioning proxy (DPP) – allowing seamless and secure data provisioning across organizations while masking the identity of upstream data sources to protect business interests. The components enable the creation of end-to-end, standards-based, flexible data value chains. We validate their applicability through a real-world financial services use case involving loan processing, which illustrates data sharing and protection challenges in B2B ecosystems.

Large Language Models (LLMs) increasingly serve as knowledge interfaces, yet systematically assessing their reliability with conflicting information remains difficult. We propose an RDF-based framework to assess multilingual LLM quality, focusing on knowledge conflicts. Our approach captures model responses across four distinct context conditions (complete, incomplete, conflicting, and no-context information) in German and English. This structured representation enables the comprehensive analysis of knowledge leakage-where models favor training data over provided context-error detection, and multilingual consistency. We demonstrate the framework through a fire safety domain experiment, revealing critical patterns in context prioritization and language-specific performance, and demonstrating that our vocabulary was sufficient to express every assessment facet encountered in the 28-question study.

Accessing knowledge via multilingual natural-language interfaces is one of the emerging challenges in the field of information retrieval and related ones. Structured knowledge stored in knowledge graphs can be queried via a specific query language (e.g., SPARQL). Therefore, one needs to transform natural-language input into a query to fulfill an information need. Prior approaches mostly focused on combining components (e.g., rule-based or neural-based) that solve downstream tasks and come up with an answer at the end. We introduce mKGQAgent, a human-inspired framework that breaks down the task of converting natural language questions into SPARQL queries into modular, interpretable subtasks. By leveraging a coordinated LLM agent workflow for planning, entity linking, and query refinement - guided by an experience pool for in-context learning - mKGQAgent efficiently handles multilingual KGQA. Evaluated on the DBpedia- and Corporate-based KGQA benchmarks within the Text2SPARQL challenge 2025, our approach took first place among the other participants. This work opens new avenues for developing human-like reasoning systems in multilingual semantic parsing.

Git-based project data holds valuable insights beyond version control, including collaboration patterns and project management details. However, the heterogeneity of Git hosting platforms complicates unified analysis. GitLotus addresses this challenge by transforming Git repositories into RDF, enabling a structured and platform-independent representation. It extracts repository data via the Git protocol, integrates metadata from GitHub while handling API limitations, and exposes results through persistent SPARQL endpoints.

In the paper, we describe the data sharing within a data-driven B2B ecosystem using Solid technologies. Using a real-world use case, we describe our approach and implementation, as well as the (previously non-existent) referencing of a purpose ontology to fully express the intended use of shared data in a machine-readable and analyzable form. We follow the intention of establishing a safe, robust, and traceable linked-data-driven B2B ecosystem based on Solid, where high standards of data privacy and thriftiness have to be met.

Knowledge Graphs rely on conceptional knowledge (TBox) and concrete data instances (ABox). Both types of statements are represented by RDF triples, hence, they are located in the same data storage. As this is an advantage from the aspect of generalization, it might become difficult to distinguish between the purpose of the triples when exploring a knowledge graph. Additionally, the success of knowledge graphs leads to larger data sets that might make it also harder to identify and recognize the information of interest, its connections, and patterns. KinGVisher was designed and developed as a visual explorer for knowledge graphs, providing automatic graph layouts and supporting users in creating a clear view of the knowledge graph. We publish this as a full-fledged, open-sourced web application online, which is freely available to all users.

Question answering is an approach to retrieving information from a knowledge base using natural language. Within question answering systems that work over knowledge graphs (KGQA), a ranked list of SPARQL query candidates is typically computed for the given natural-language input, where the top-ranked query should reflect the intention and semantics of the given user’s question. This article follows our long-term research agenda of providing trustworthy KGQA systems by presenting an approach for filtering incorrect queries. Here, we employ (large) language models (LMs/LLMs) to distinguish between correct and incorrect queries. The main difference to the previous work is that we address here multilingual questions represented in major languages (English, German, French, Spanish, and Russian), and confirm the generalizability of the approach by also evaluating it on some low-resource languages (Ukrainian, Armenian, Lithuanian, Belarusian, and Bashkir). The considered LMs (BERT, DistilBERT, Mistral, Zephyr, GPT-3.5, and GPT-4) were applied to the KGQA systems – QAnswer (real-world system) and MemQA (idealized system) – as SPARQL query filters. The approach was evaluated on the multilingual dataset QALD-9-plus, which is based on the Wikidata knowledge graph. The experimental results imply that the considered KGQA systems achieve quality improvements for all languages when using our query-filtering approach.

Question Answering systems working over Knowledge Graphs (KGQA) generate a ranked list of SPARQL query candidates for a given natural-language question. In this paper, we follow our long-term research agenda of providing trustworthy KGQA systems – here – by presenting a query filtering approach that utilizes (large) language models (LMs/LLMs), s.t., correct and incorrect queries can be distinguished. In contrast to the previous work, we address here multilingual questions represented in major languages (English, German, French, Spanish, and Russian), and confirm the generalizability of our approach by also evaluating it on low-resource languages (Ukrainian, Armenian, Lithuanian, Belarusian, and Bashkir). For our experiments, we used the following LMs: BERT, DistilBERT, Mistral, Zephyr, GPT-3.5, and GPT-4. The LMs were applied to the KGQA systems – QAnswer and MemQA – as SPARQL query filters. The approach was evaluated on the multilingual Wikidata-based dataset QALD-9-plus. The experimental results suggest that the KGQA systems achieve quality improvements for all languages when using our query-filtering approach.