Large language models (LLMs) are increasingly used to generate natural language explanations of predictive machine learning models, yet their outputs often emphasize plausibility over faithfulness to the underlying model logic. This study applies counterfactual testing to evaluate the fidelity of LLM explanations in a structured setting: decision tree models predicting customer churn (i.e., when customers stop using a company’s product or service-a classic benchmark problem in predictive modeling research). We design two counterfactual tests across four prompting strategies that vary in whether dataset and/or model information are provided. Results from 1,200 generated explanations show that supplying model context substantially improves faithfulness whereas dataset context provides little benefit, and LLMs often broaden explanations beyond the model’s true decision path. Notably, even when given full model logic, LLMs remain predisposed toward generating plausible rather than strictly faithful explanations. These findings highlight both the promise and the limits of LLMs as interpreters of predictive models and introduce counterfactual testing as a potential framework for evaluating explanation faithfulness in data-heavy contexts.
Educational institutions lack a systematic understanding of how generative AI policies are implemented across computing courses, and keyword-based methods alone do not capture their nuance and complexity. This study presents a syllabus-based framework for analyzing generative AI policy adoption practices and trends in computing education. Policyrelated text is identified through keyword-based retrieval with contextual expansion and classified into policy types using a combination of rule-based inference and embedding-based semantic similarity. The resulting policy categories (No Policy, Permitted, Restricted, Discouraged, and Prohibited), together with extracted course metadata, enable analysis of adoption patterns over time and across course levels. Using 148 course syllabi from Spring 2024 to Fall 2025, results show that adoption increased from $\mathbf{12.8 \%}$ to $\mathbf{58.8 \%}$ over four semesters, with similar adoption rates between undergraduate $(40.4 \%)$ and graduate $(39.6 \%)$ courses. Instructor concerns are identified via semantic matching, with primary emphasis on over-dependence $(72.9 \%)$, proper citation $(57.6 \%)$, and appropriate use $(52.5 \%)$. This work provides a validated methodology ($90.56 \%$ accuracy) for characterizing generative AI policy adoption in higher education.
In large and complex systems such as the Circular Factory, vast amounts of heterogeneous and uncertain data are generated. Classical Bayesian Networks (BNs) provide a principled framework for reasoning under uncertainty, but their reliance on discrete Conditional Probability Tables (CPTs) limits their applicability in continuous domains. In the context of semantic computing, where heterogeneous sensor data and knowledge graph information must be integrated, this limitation prevents probabilistic reasoning from being consistently applied to continuous quantities and measurement-based information. This paper introduces a framework for continuous Bayesian Networks that replaces discrete CPTs with functional dependencies and integrates the Guide to the Expression of Uncertainty in Measurement (GUM) as a standardized approach for analytical uncertainty propagation. The proposed method enables probabilistic reasoning over continuous variables while explicitly accounting for measurement and parameter uncertainty in a mathematically transparent way. To complement the analytical formulation, Monte Carlo (MC) simulations with Gaussian Mixture Models (GMMs) are employed to handle non-linear and non-Gaussian cases, allowing for numerical validation of the GUM approximation. We present illustrative examples comparing both approaches and discuss their agreement and limitations. The results demonstrate that GUM-based inference provides accurate and computationally efficient uncertainty estimates in locally linear regions, while Monte Carlo methods serve as a reliable benchmark for more complex, non-linear dependencies. This framework thus lays the foundation for robust and interpretable uncertainty reasoning in environments such as the Circular Factory. Furthermore, it provides a bridge between probabilistic reasoning and semantic representations by enabling continuous uncertainty information to be embedded within knowledge-based systems, thus advancing uncertainty-aware semantic computing.
Enterprise Knowledge Graphs have become essential for unifying heterogeneous data and enforcing semantic governance. However, the construction of their underlying ontologies remains a resource-intensive, manual process that relies heavily on domain expertise. This paper introduces OntoEKG, a LLM-driven pipeline designed to accelerate the generation of domain-specific ontologies from unstructured enterprise data. Our approach decomposes the modelling task into two distinct phases: an extraction module that identifies core classes and properties, and an entailment module that logically structures these elements into a hierarchy before serialising them into standard RDF. Addressing the significant lack of comprehensive benchmarks for end-to-end ontology construction, we adopt a new evaluation dataset derived from documents across the Data, Finance, and Logistics sectors. Experimental results highlight both the potential and the challenges of this approach, achieving a fuzzy-match F1-score of 0.724 in the Data domain while revealing limitations in scope definition and hierarchical reasoning.
Traditional task-oriented dialog (ToD) systems rely heavily on labor-intensive turn-level annotations, such as dialogue states and policy labels, for training. This work explores whether large language models (LLMs) can be fine-tuned solely on natural language dialogs to perform ToD tasks, without requiring such annotations. We evaluate their ability to generalize to unseen domains and compare their performance with models trained on fully annotated data. Through extensive experiments with three open-source LLMs of varying sizes and two diverse ToD datasets, we find that models fine-tuned without turn-level annotations generate coherent and contextually appropriate responses. However, their task completion performance - measured by accurate execution of API calls - remains suboptimal, with the best models achieving only around 53 propose ZeroToD, a framework that incorporates a schema augmentation mechanism to enhance API call accuracy and overall task completion rates, particularly in out-of-domain settings. We also compare ZeroToD with fine-tuning-free alternatives, such as prompting off-the-shelf LLMs, and find that our framework enables smaller, fine-tuned models that outperform large-scale proprietary LLMs in task completion. Additionally, a human study evaluating informativeness, fluency, and task completion confirms our empirical findings. These findings suggest the feasibility of developing cost-effective, scalable, and zero-shot generalizable ToD systems for real-world applications.
Language Models (LMs) have shown their application for tasks pertinent to code and several code~LMs have been proposed recently. The majority of the studies in this direction only focus on the improvements in performance of the LMs on different benchmarks, whereas LMs are considered black boxes. Besides this, a handful of works attempt to understand the role of attention layers in the code~LMs. Nonetheless, feed-forward layers remain under-explored which consist of two-thirds of a typical transformer model's parameters. In this work, we attempt to gain insights into the inner workings of code language models by examining the feed-forward layers. To conduct our investigations, we use two state-of-the-art code~LMs, Codegen-Mono and Ploycoder, and three widely used programming languages, Java, Go, and Python. We focus on examining the organization of stored concepts, the editability of these concepts, and the roles of different layers and input context size variations for output generation. Our empirical findings demonstrate that lower layers capture syntactic patterns while higher layers encode abstract concepts and semantics. We show concepts of interest can be edited within feed-forward layers without compromising code~LM performance. Additionally, we observe initial layers serve as ``thinking'' layers, while later layers are crucial for predicting subsequent code tokens. Furthermore, we discover earlier layers can accurately predict smaller contexts, but larger contexts need critical later layers' contributions. We anticipate these findings will facilitate better understanding, debugging, and testing of code~LMs.
Semantic image segmentation enables computing systems to understand the semantic patterns of image pixels by using deep learning models to classify the pixels into specific labels. The deep-learning models' performance in image classification has been evaluated by comparing the predicted images using deep-learned features with human-labeled images or mask images. However, there remains a substantial need to investigate the performance of machine learning models that do not use deep-learned features but use hand-crafted features. In this paper, we perform a comprehensive evaluation of the performance of the eight machine learning models using 46 hand-crafted features extracted from the PanNuke dataset including 5,179 hematoxylin and eosin images with 161,739 cell nuclei, by optimizing feature selection through Bayesian optimization. The evaluation results indicate that the ensemble learning-based models achieve higher performance compared to others across precision, recall, f1-score, and accuracy.
The interactive nature of Large Language Models (LLMs), which closely track user data and context, has prompted users to share personal and private information in unprecedented ways. Even when users opt out of allowing their data to be used for training, these privacy settings offer limited protection when LLM providers operate in jurisdictions with weak privacy laws, invasive government surveillance, or poor data security practices. In such cases, the risk of sensitive information, including Personally Identifiable Information (PII), being mishandled or exposed remains high. To address this, we propose the concept of an “LLM gatekeeper”, a lightweight, locally run model that filters out sensitive information from user queries before they are sent to the potentially untrustworthy, though highly capable, cloud-based LLM. Through experiments with human subjects, we demonstrate that this dual-model approach introduces minimal overhead while significantly enhancing user privacy, without compromising the quality of LLM responses.
Although aligned with security-based principles, Blockchain networks have maintained some exposure to fraudulent transactions. This paper introduces a novel methodology and framework for effectively characterizing fraud within blockchain networks and a method for prevention. To leverage the transparency of the blockchain, suitable starting data can be acquired to characterize potentially nefarious transactions. The framework presented applies generative AI at two levels: to support the characterization of synthetic training data for scenarios that may yet be deployed and to generate suitable testing scenarios for constructing effective techniques to safeguard transactions.
The surge in deepfake technology poses significant threats to the integrity of digital media. This paper combines generative AI with Blockchain - based authentication for a novel approach to deepfake detection and media verification. By pairing a Generative Adversarial Model (GAN) for synthetic data verification with Blockchain's immutability a secure and robust framework for authentication is determined. The first step of this process is to characterize the method used to forge unauthentic media and leverage it with a discriminative model for advanced pattern detections. The second step would involve creating unique digital fingerprints for media on the blockchain developing characterizations of authentic as well as unauthentic media. Preliminary results show that this dual-layered system effectively enhances digital content's reliability and security.
While Two-factor Authentication (2FA) has significantly improved authentication security, it is now facing an increasing range of sophisticated attacks. To address this challenge, we propose augmenting 2FA with unique environmental signatures found in the user's local surroundings. Since most attackers operate remotely, these environmental factors can serve as an effective barrier against unauthorized access. Our approach leverages custom lighting through smart bulb technology, which is becoming increasingly common in households, to generate dynamic color patterns as an additional, non-interactive authentication factor. We developed a prototype where these color patterns are detected and analyzed for authentication. Our evaluation examines both the system's effectiveness and usability in everyday settings. Results demonstrate that this method significantly enhances security while maintaining user convenience, presenting a promising solution for the next generation of multi-factor authentication frameworks. This work highlights the potential of integrating smart home technology to create secure and user-friendly authentication practices.
We propose a method, Curvature-Aware Fairness in Attention (CAFA), for mitigating demographic biases in transformer-based language models. By viewing the attention space as a Riemannian manifold, CAFA adjusts the metric tensor and Ricci curvature to reshape how tokens interact. Our analysis proves that these geometric modifications alter geodesic paths, driving attention away from stereotypical associations. Curvature modifications constrain the spread of biased signals by influencing the manifold's geodesic flow, resulting in a more balanced attention distribution across diverse contexts. Empirical evaluations on the StereoSet benchmark show a notable reduction in stereotype scores, with only a minimal impact on language modeling performance.
The growing complexity of digital assemblies presents significant challenges, particularly in understanding their characteristics and interdependencies. Traditional approaches, which focus largely on geometric properties, fall short in capturing the functional dependencies between components. This research introduces a novel approach using semantic enrichment of CAD components and Graph Neural Networks (GNNs) to classify mechanical parts and analyze their interconnections. By leveraging open-source gearbox designs which are based on native CAD data in addition to STEP data, the methodology showcases the ability of graph-based structures to account for both geometric and functional relationships, providing a more comprehensive digital understanding of assemblies.
This paper introduces a hybrid approach for the text-to-SPARQL task involving Large Language Models (LLMs) by integrating Retrieval-Augmented Generation (RAG) with fewshot learning techniques. The approach enhances the AutoKGQA framework by incorporating query examples into the LLM while selecting minimal knowledge graph (KG) subgraphs as context, thereby improving its ability to interpret and answer natural language queries. Experimental results on the SciQA benchmark indicate that this integration increased the F1-score by 0.16. Notably, the framework excelled in the zero-shot setting with an F1-score of 0.73, significantly outperforming the prior score of 0.26. Additionally, the paper introduces an automated procedure to extract a minimal T-Box from KGs lacking an explicit schema, optimizing query processing by limiting deep neighborhood exploration. These findings suggest that combining KG context with query examples is an effective strategy, particularly for developing generalizable systems.
The swift progress of digital technologies and recent global pandemic events have intensified the focus on leveraging these technologies to improve health care service delivery and workflow, particularly during crises. Action plans that consolidate existing digital transformation programs are being re-viewed to establish core infrastructure for sustainable health care solutions. For instance, reforming health and social care to personalize home care can reduce the need for treatment in overcrowded acute hospital settings, improving experiences and outcomes for both health care professionals and service users. In this data-driven field, tackling the interoperability challenge through standards-based roadmaps is essential for establishing seamless connections between health and social care services. This strategy ensures secure and reliable data exchange across various healthcare providers. In this paper, we propose a methodology for extracting, transforming, and loading data through a semi-automated process, utilizing the Common Semantic Standardized Data Model (CSSDM) to build personalized healthcare knowledge graphs (KGs). The CSSDM is based on the formal ontology of ISO 13940 ContSys and integrates FHIR-based specifications to support the structural attributes necessary for KG creation. We propose that the CSSDM facilitates data harmonization and linking, offering an alternative approach to interoperability. This approach promotes a novel form of collaboration between companies developing health information systems and cloud-enabled health services. Consequently, it provides multiple stakeholders with access to high-quality data and information sharing.
Modern video-based eye trackers have reached a level of accuracy sufficient for gaze-based human-computer interaction. In many but not all applications, accuracy even allows for replacing a computer mouse with gaze-based pointing. The Multi-Modal Interaction Concept for Efficient Input (M2ice) tackles accuracy problems with an on-demand hybrid linear-fisheye magnifier. This paper explores how semantic analysis of screen content can be used as part of an alternative solution to the accuracy problem. This solution integrates machine-learning-based near real-time image analysis into the M2ice interaction concept and adds semantic highlighting to screen elements near the user's interaction point of gaze. Semantic highlighting identifies clickable screen elements and adds a number as well as a bounding box to each of these elements. This allows users to select elements with a keystroke. It thus combines visual magnification with visual semantic enrichment. The image analysis is based on a machine learning model specifically trained for the respective use case. A user study (n = 20) showed no significant speed difference with the use of semantic highlighting but a 94.3% decrease in interaction errors (misclicks) with the use of semantic highlighting compared to the traditional selection and magnification approaches.
Machine Learning (ML) models are essential across numerous fields, including those where safety is paramount, e.g., autonomous driving, which is the primary focus of this paper. In many areas, the opaque nature of ML systems may be a minor issue, but in safety-critical contexts, it creates challenges in establishing trust. To fully harness ML models in these high-stakes fields, a method is needed to enhance confidence in their reliability and accuracy without the need for human oversight on every decision. This research introduces a technique aimed at boosting trust in ML models by improving the robustness and thoroughness of the training datasets. Since ML models are shaped by the data they are trained on, ensuring the dataset's completeness can potentially enhance confidence in the model's performance. To achieve this, the paper suggests leveraging both a domain-specific ontology and an image quality ontology to assess the coverage and robustness of the dataset. A case study is presented within the context of emergency road vehicles.
The incorporation of computational workflows is well documented in disciplines like astrophysics, medicine, and environmental sciences. Their ability to represent computational components and data interactions enables automated scientific workflows, leading to increased efficiency and reproducibility. Diverse approaches have been proposed to improve the experience of crafting scientific experiments into computational workflows where manual and semi-automatic methods are well documented. In previous work, we proposed a method for automating workflow composition. However, component-to-component integration remains a challenge as common metadata notations are not enough to automatically identify equivalent variables under implicit scientific assumptions. In this work, we present VaR-O, an ontology to semantically identify equivalent variables based on implicit design decisions taken by domain experts when building scientific components. We also implement a tie-breaking strategy for component selection during the automated workflow composition process.
Tables have become a ubiquitous standard for capturing, storing, and sharing data on the web. This is primarily due to the semi-structured nature of tables, where relationships between data are often ambiguously encoded using locality. While this format can be easy for humans to interpret in simple cases, as table complexity increases, so does the difficulty in interpretability. To bridge this context gap, many data publishers provide a data dictionary to capture schema elements' meaning through text descriptions. Existing work compounds the need for data dictionaries to improve tabular interoperability, but few provide detailed requirements for data dictionary descriptions. This paper identifies and defines three common types of data dictionary descriptions in the biomedical domain. We then compare the effectiveness of each description type by normalizing data dictionary descriptions to a single type using large language models and measuring their performance using a semantic tabular interpretation algorithm. Our experiments show that intensional descriptions, which describe the general properties a column member should have, are most effective for tabular alignment and improve the reusability of data dictionaries.
The use of Large Language Models (LLMs) in the healthcare domain is one of the most interesting research topics in recent years. In this position paper, we report some of the main issues and approaches in the literature and outline the possibility of integrating an LLM conversational module into a vital parameters monitoring tool. In fact, the use of an LLM in such a tool could improve both the effectiveness of the system and user engagement.
