The central bottleneck in computational geothermal science is not simulator fidelity or data scarcity—it is the abstraction itself. Geothermal energy is increasingly important to the clean energy transition, yet its computational core still follows a legacy simulate-then-optimize paradigm: a deterministic simulator is calibrated to sparse observations and then used to optimize decisions within a fixed model. Hidden inside this pipeline are three commitments—one predicted future, one mostly static operating strategy, and one fitted model per site. We argue that, for next-generation enhanced geothermal systems, the subsurface is partially observed, heterogeneous, and intervention-sensitive, and the information available to characterize it is limited. As a result, forecasting and decision-making must reason over multiple physically plausible futures under uncertainty. Our central claim is that geothermal should be reframed as an adaptive problem of inference, intervention, and discovery. Under this view, simulation becomes conditional generation over plausible reservoir futures rather than point prediction of one trajectory. Operation becomes adaptive decision making over belief states rather than offline scheduling under a presumed known state. Calibration becomes the separation of transferable physical structure from site-specific corrections rather than repeated fitting within a fixed equation class. These are not three independent engineering problems; they are three phases of a single inference cycle. This reframing matters because, in geothermal, uncertainty is not merely something to quantify; it is something operations act upon and reshape. Likewise, persistent model mismatch is not merely an engineering nuisance to suppress; it is the primary scientific signal from which missing or site-modulated physics can be discovered. We therefore organize the paper around three consequences of this reframing: generative world models of reservoir evolution, belief-state policy learning for sustainable operation, and data-to-equation discovery for transferable geophysics. Taken together, these directions define a new agenda for geothermal AI beyond faster surrogate prediction toward adaptive subsurface intelligence where inference, intervention, and discovery are intrinsically coupled.
Multimodal deep learning has achieved remarkable progress by leveraging complementary information across heterogeneous data sources such as texts, images, audios, and structured signals. While increasingly powerful encoders and fusion mechanisms have improved predictive performance, the reliability of multimodal systems remains a critical challenge. In particular, modality disagreement, distribution shifts, and noisy inputs can lead to overconfident yet incorrect predictions. Distinct from existing surveys on uncertainty in deep learning [45] or on multimodal learning [8; 135], this survey jointly covers three aspects: (i) the structural foundations of multimodal classification examined through the lens of the un- certainty challenges each design choice introduces; (ii) un- certainty quantification in both unimodal and multimodal settings; and (iii) set-valued classification as a decision-level strategy for cautious multimodal prediction. We first review foundational aspects of multimodal representation learning and fusion strategies, highlighting their structural limitations in modeling inter-modal dependence and the uncertainty challenges each stage introduces. We then examine uncertainty quantification methods in deep learning, including both probabilistic and evidence-theoretic approaches, and analyze how these techniques extend to multimodal settings. Special attention is given to conflict-aware fusion mechanisms and to decision-level strategies such as set- valued classification, which enable more cautious and informative predictions. Beyond reviewing existing methods, we identify key open challenges, including the modeling of partial dependence be- tween modalities, the need for systematic benchmarking of multimodal uncertainty, and the integration of uncertainty into decision-making pipelines. Finally, we discuss how these reliability challenges extend to emerging multimodal agentic systems. By synthesizing advances across multimodal learning and uncertainty modeling, this survey aims to provide a unified perspective and to outline recent research directions toward more reliable multimodal AI systems.
Large language models (LLMs) and large vision-language models (LVLMs) have demonstrated impressive generative capabilities, yet ensuring their outputs align with user intent is still challenging. While most existing approaches address this issue at the training stage, inference-time approaches like decoding methods offer a more efficient and scalable solution. Decoding methods control model generation by guiding token-level selection, performing sequencelevel generation, or generating tokens in parallel to accelerate the process. In this survey, we identify three emerging paradigms from recent works on decoding methods for LLMs and LVLMs, provide a systematic review of these methods, highlight ongoing challenges, and discuss potential future research directions. Our goal is to underscore the efficiency and effectiveness of decoding methods and offer a practical view of their applications. Paper lists and more resources on decoding methods for LLMs and LVLMs can be found at https://github.com/wang2226/Awesome-LLM-Decoding.
A challenge in mitigating social bias in fine-tuned language models (LMs) is the potential reduction in language modeling capability, which can harm downstream performance. Counterfactual data augmentation (CDA), a widely used method for fine-tuning, highlights this issue by generating synthetic data that may align poorly with real-world distributions or creating overly simplistic counterfactuals that ignore the social context of altered sensitive attributes (e.g., gender) in the pretraining corpus. To address these limitations, we propose a simple yet effective context-augmented CDA method, Context-CDA, which uses large LMs to enhance the diversity and contextual relevance of the debiasing corpus. By minimizing discrepancies between the debiasing corpus and pretraining data through augmented context, this approach ensures better alignment, enhancing language modeling capability. We then employ uncertainty-based filtering to exclude generated counterfactuals considered low-quality by the target smaller LMs (i.e., LMs to be debiased), further improving the fine-tuning corpus quality. Experimental results on gender bias benchmarks demonstrate that Context-CDA effectively mitigates bias without sacrificing language modeling performance while offering insights into social biases by analyzing distribution shifts in next-token generation probabilities.
Large language models (LLMs) are trained on massive corpora that may contain sensitive information, creating privacy risks under membership inference attacks (MIAs). Knowledge distillation is widely used to compress LLMs into smaller student models, but its privacy implications are poorly understood. We systematically evaluate how distillation affects MIA vulnerability across six teacher-student model pairs and six attack methods. We find that distilled student models do not consistently exhibit lower MIA success than their teacher models, and in some cases demonstrate substantially higher member-specific attack success, challenging the assumption that knowledge distillation inherently improves privacy. We attribute this to mixed supervision in distillation: for vulnerable training data points, teacher predictions often align with ground-truth labels, causing student models to learn overly confident predictions that amplify the separability between members and non-members; conversely, for non-vulnerable points, teacher predictions and ground truth frequently diverge, providing inconsistent learning signals. To mitigate this, we propose three practical interventions – restricting distillation to non-vulnerable points, adding a low-dimensional Bottleneck Projection, and a normalization variant (NoNorm). Experiments show these methods reduce both aggregate and member-specific MIA success while preserving model utility, improving privacy-utility trade-offs for distilled LLMs.
The rapid advancements in large Language models (LLMs) have significantly enhanced their reasoning capabilities, driven by various strategies such as multi-agent collaboration. However, unlike the well-established performance improvements achieved through scaling data and model size, the scaling of reasoning in LLMs is more complex and can even negatively impact reasoning performance, introducing new challenges in model alignment and robustness. In this survey, we provide a comprehensive examination of scaling in LLM reasoning, categorizing it into multiple dimensions and analyzing how and to what extent different scaling strategies contribute to improving reasoning capabilities. We begin by exploring scaling in input size, which enables LLMs to process and utilize a more extensive context for improved reasoning. Next, we analyze scaling in reasoning steps that improve multi-step inference and logical consistency. We then examine scaling in reasoning rounds, where iterative interactions refine reasoning outcomes. Furthermore, we discuss scaling in training-enabled reasoning, focusing on optimization through iterative model improvement. Finally, we outline future directions for further advancing LLM reasoning. By synthesizing these diverse perspectives, this survey aims to provide insights into how scaling strategies fundamentally enhance the reasoning capabilities of LLMs and further guide the development of next-generation AI systems.
The surge of data available on the Internet has driven the adoption of a wide range of computational methods for analyzing and extracting insights from large-scale data. Among these, Machine Learning (ML) has become a central paradigm, offering powerful tools for pattern discovery, prediction, and representation learning across many domains. At the same time, real-world data often exhibit properties such as noise, imbalance, sparsity, limited supervision, and high dimensionality, motivating the use of additional analytical perspectives that can complement standard ML pipelines. One such perspective is Topological Data Analysis (TDA), a statistical framework that focuses on the intrinsic shape and structural organization of data. Rather than replacing ML, TDA offers a complementary lens for characterizing geometric and topological properties that may be difficult to capture with conventional feature-based or purely predictive approaches. This has motivated a growing body of work that integrates TDA into ML workflows, particularly in settings where data structure plays an important role. Despite this promise, TDA has received relatively limited attention in Natural Language Processing (NLP) compared to domains with more overt structural regularities, such as computer vision. Nevertheless, a dedicated community of researchers has explored its use in NLP, leading to 137 papers that we comprehensively survey in this work. We organize these studies into theoretical and nontheoretical approaches. Theoretical approaches use topology to explain linguistic phenomena, whereas non-theoretical approaches incorporate TDA into ML-based pipelines through a variety of numerical representations. We conclude by discussing the key challenges and open questions that continue to shape this emerging area. Resources and a list of papers are available at: https://github.com/AdaUchendu/AwesomeTDA4NLP.
Large language models (LLMs) have shown re2 markable performance in tackling natural language 3 tasks, yet the internal mechanisms that enable their 4 impressive generalization and reasoning abilities 5 remain opaque. This lack of transparency presents 6 significant challenges in fundamentally eliminating 7 undesirable behaviors such as hallucinations and 8 toxicity, hindering the safe and beneficial deploy9 ment of LLMs. This survey paper aims to un10 cover the internal working mechanisms underly11 ing LLM functionality through the lens of explain12 ability. First, we review how knowledge is en13 coded within LLMs via mechanistic interpretabil14 ity techniques. Then, we summarize what knowl15 edge is embedded in LLMrepresentations by lever16 aging probing techniques and representation engi17 neering. Additionally, we investigate the training 18 dynamics to explore models' generalization abili19 ties through grokking and memorization. Finally, 20 we explore how the insights gained from these ex21 planations can further enhance LLM performance 22 through model editing, improve efficiency through 23 pruning, and better align with human values.
Large language models have demonstrated remarkable capabilities in text generation and problem solving, yet they continue to face fundamental challenges such as hallucination, lack of factual grounding, and limited reasoning reliability. The core of these issues lies the question of how LLMs acquire and use knowledge. While internal knowledge embedded in model parameters enables impressive generalization, it is often insu!cient for up-to-date or domain-speci''c tasks. External knowledge integration, such as retrieval-augmented generation (RAG), provides grounding and factuality but introduces challenges of retrieval quality, latency, and reliability. Beyond these paradigms, recent advances in agentic LLMs extend models from passive generators to active problem solvers that can reason, plan, and interact with external tools. This survey provides a uni''ed, knowledge-centric perspective on LLMs, organized along three complementary dimensions: (i) reactive: internal knowledge, (ii) lightly-active: external knowledge, and (iii) proactive: agentic knowledge utilization for reasoning and tool interaction. We provide a taxonomy of knowledge usage in LLMs, analyze their respective strengths and limitations, and highlight how these paradigms interact in real-world systems. Finally, we identify open challenges to facilitate future research.
Large Language Models (LLMs) exhibit exceptional proficiency in comprehending human language. Despite their significant success across a wide array of tasks, understanding tabular data remains a challenging task. Especially, tabular data lacks an intrinsic order of the different features (table fields), whereas LLMs take only sequential inputs. Consequently, an artificial order is imposed, the impact of which on the performance of LLMs has not yet been thoroughly investigated. Surprisingly, as discovered in this work, this artificially induced order bias dramatically influences the performance of LLMs on tasks related to tabular data. Mitigating the order bias presents a significant challenge. To address this, we propose a simple and cost-effective method, Re-Ordering Tabular feATures fOR LLM (ROTATOR-LLM), to conduct test-time compute without fine-tuning the base LLM. Aiming at optimizing the feature order of tabular data and boosting LLMs' capability to better understand the data semantics, ROTATOR-LLM re-frames the ordering problem as a feature trajectory generation task. A dynamic programming based meta-controller is trained to auto-regressively generate an individualized feature trajectory for each data instance via accumulative value estimation of the serialized feature input through the LLM's final performance metrics. Model performance is maximized by iteratively selecting features across different steps. Experimental results on multiple datasets and LLMs show close to or over 20% performance boosts via features reordered by ROTATOR-LLM against the un-ordered counterpart. Meanwhile, it outperforms stateof- the-Art tabular LLM methods with significant margin.
This document aims to provide an overview and synopsis of frugal AI, with a particular focus on its role in promoting cost-effective and sustainable innovation in the context of limited resources. It discusses the environmental impact of AI technologies and the importance of optimising AI systems for efficiency and accessibility. It explains the interface between AI, sustainability and innovation. In fourteen sections, it also makes interested readers aware of various research topics related to frugal AI, raises open questions for further exploration, and provides pointers and references.
The popular problem of Influence Maximization (IM) asks for the k users who can maximize the influence of a fixed post in a social network. In contrast, the problem of Content- Aware Influence Maximization (CAIM) asks for the k features to form a viral tunable post in a social network starting its diffusion from a fixed set of initial adopters. CAIM paves the way for a number of novel problems to be studied that altogether can lead to the development of a system that would be valuable for advertisers who manage social network pages. This holds since features (brands) in CAIM map to specific social network pages and each advertiser of a certain page can utilize their own feature along with others in a variety of ways to form a proper content for influence and subscription maximization purposes. In this article, we present our content-based perspective about how such a system (named b2biers) can be built, the technical challenges about it, and the novel services that it can yield to every kind of brands and advertisers running the brand pages.
Users often navigate multiple platforms online, each characterized by its own set of scarce data. Recommender systems face a significant challenge in such fragmented environments. This paper proposes a novel approach to enhance recommendation systems by leveraging connections across distinct yet conceptually similar datasets from multiple platforms. We introduce a unique scenario of dual-target overlapping-free cross-platform recommendation, presenting a bridging mechanism to mutually improve across platforms and learn latent user preferences. Our approach addresses the data sparsity prevalent in each platform and enhances recommendation quality by harnessing redundant, rich, and similar domain data. Experiments validate the effectiveness of our method, demonstrating substantial improvements in recommendation quality.
Information Extraction (IE) encompasses a diverse array of tasks in Natural Language Processing (NLP), including Named Entity Recognition (NER), Entity Linking (EL), and Attribute Value Extraction (AVE), all of which aim to derive structured representations from unstructured text. Despite their shared goals, these tasks are often studied in isolation, leading to redundant methods and fragmented advances. This work introduces a unified framework for IE centered on the concept of a Span: a contiguous sequence of tokens annotated with one or more semantic labels. By adopting spans as the foundational unit of analysis, we unify a broad class of IE tasks under a common formulation. We formalize key dimensions of this span-oriented paradigm: task formulation via span identification and classification, evaluation through boundary- and label-sensitive metrics, representation strategies including token-to-span encodings, architectural components tailored to span extraction, and modeling strategies grounded in pre-trained language models. We also identify persistent challenges, such as boundary ambiguity and context-label mismatch, that cross tasks. Through this unified lens, we synthesize and standardize a field long divided by task-specific assumptions. The result is a cohesive framework that supports cross-task generalization, standardized evaluation, and principled model design, which lays the foundation for future advances in both spanbased and general-purpose information extraction systems.
The threat that online fake news and misinformation pose to democracy, justice, public confidence, and especially to vulnerable populations has led to a sharp increase in the need for fake news detection and intervention. Whether multi-modal or pure text-based, most existing fake news detection methods depend on textual analysis of entire articles. However, these fake news detection methods come with certain limitations. For instance, fake news detection methods that rely on full text can be computationally inefficient, demand large amounts of training data to achieve competitive accuracy, and may lack robustness across different datasets. This is because fake news datasets have strong variations in terms of the level and types of information they provide; where some can include large paragraphs of text with images and metadata, and others can be a few short sentences. Perhaps if one could only use minimal information to detect fake news, fake news detection methods could become more robust and resilient to the lack of information. We aim to overcome these limitations by detecting fake news using systematically selected, limited information that is both effective and capable of delivering robust, promising performance. We propose a framework called SLIM (Systematically-selected Limited Information) for fake news detection. In SLIM, we quantify the amount of information by introducing information-theoretic measures. SLIM leverages limited information (e.g., a few named entities) to achieve performance in fake news detection comparable to that of state-of-the-art obtained using the full text, even when the dataset is sparse. Furthermore, by combining various types of limited information, SLIM can perform even better while significantly reducing the quantity of information required for training compared to state-of-the-art language model-based fake news detection techniques.
Large language models (LLMs) deliver superior performance but require substantial computational resources and operate with relatively low efficiency, while smaller models can efficiently handle simpler tasks with fewer resources. LLM routing is a crucial paradigm that dynamically selects the most suitable large language models from a pool of candidates to process diverse inputs, ensuring optimal resource utilization while maintaining response quality. Existing routing frameworks typically model this as a locally optimal decision-making problem, selecting the presumed best-fit LLM for each query individually, which overlooks global budget constraints, resulting in ineffective resource allocation. To tackle this problem, we introduce OmniRouter, a fundamentally controllable routing framework for multi-LLM serving. Instead of making per-query greedy choices, OmniRouter models the routing task as a constrained optimization problem, assigning models that minimize total cost while ensuring the required performance level. Specifically, a hybrid retrieval-augmented predictor is designed to predict the capabilities and costs of LLMs. After obtaining the predicted cost and performance, we utilize a constrained optimizer for cost-optimal assignments that employs Lagrangian dual decomposition with adaptive multipliers. It iteratively converges toward the globally optimal query-model allocation, dynamically balancing latency minimization against quality thresholds while adhering to heterogeneous capacity constraints. Experiments show that OmniRouter achieves up to 6.30
Pre-trained language models (PLMs) have achieved impressive results on various natural language processing tasks. However, recent research has revealed that these models often rely on superficial features and shortcuts instead of developing a genuine understanding of language, especially for natural language understanding (NLU) tasks. Consequently, the models struggle to generalize to out-of-domain data. In this work, we propose Divergence Based Regularization (DBR) to mitigate this shortcut learning behavior. Our method measures the divergence between the output distributions for original examples and examples where shortcut tokens have been masked. This process prevents the model's predictions from being overly influenced by shortcut features or biases. We evaluate our model on three NLU tasks and find that it improves out-of-domain performance with little loss of in-domain accuracy. Our results demonstrate that reducing the reliance on shortcuts and superficial features can enhance the generalization ability of large pre-trained language models.
Knowledge graph reasoning is pivotal in various domains such as data mining, artificial intelligence, the Web, and social sciences. These knowledge graphs function as comprehensive repositories of human knowledge, facilitating the inference of new information. Traditional symbolic reasoning, despite its strengths, struggles with the challenges posed by incomplete and noisy data within these graphs. In contrast, the rise of Neural Symbolic AI marks a significant advancement, merging the robustness of deep learning with the precision of symbolic reasoning. This integration aims to develop AI systems that are not only highly interpretable and explainable but also versatile, effectively bridging the gap between symbolic and neural methodologies. Additionally, the advent of large language models (LLMs) has opened new frontiers in knowledge graph reasoning, enabling the extraction and synthesis of knowledge in unprecedented ways. This survey offers a thorough review of knowledge graph reasoning, focusing on various query types and the classification of neural symbolic reasoning. Furthermore, it explores the innovative integration of knowledge graph reasoning with large language models, highlighting the potential for groundbreaking advancements. This comprehensive overview is designed to support researchers and practitioners across multiple fields, including data mining, AI, the Web, and social sciences, by providing a detailed understanding of the current landscape and future directions in knowledge graph reasoning.
In the rapidly evolving field of artificial intelligence, large language models (LLMs) have emerged as powerful tools for a myriad of applications, from natural language processing to decision-making support systems. However, as these models become increasingly integrated into societal frameworks, the imperative to ensure they operate within ethical and moral boundaries has never been more critical. This paper introduces a novel benchmark designed to measure and compare the moral reasoning capabilities of LLMs. We present the first comprehensive dataset specifically curated to probe the moral dimensions of LLM outputs, addressing a wide range of ethical dilemmas and scenarios reflective of real-world complexities. The main contribution of this work lies in the development of benchmark datasets and metrics for assessing the moral identity of LLMs, which accounts for nuance, contextual sensitivity, and alignment with human ethical standards. We publicly release the benchmark datasets1 and also open-source the code of the project2.
Generative AI is becoming increasingly prevalent in creative fields, sparking urgent debates over how current copyright laws can keep pace with technological innovation. Recent controversies of AI models generating near-replicas of copyrighted material highlight the need to adapt current legal frameworks and develop technical methods to mitigate copyright infringement risks. This task requires understanding the intersection between computational concepts such as large-scale data scraping and probabilistic content generation, legal definitions of originality and fair use, and economic impacts on IP rights holders. However, most existing research on copyright in AI takes a purely computer science or law-based approach, leaving a gap in coordinating these approaches that only multidisciplinary efforts can effectively address. To bridge this gap, our survey adopts a comprehensive approach synthesizing insights from law, policy, economics, and computer science. It begins by discussing the foundational goals and considerations that should be applied to copyright in generative AI, followed by methods for detecting and assessing potential violations in AI system outputs. Next, it explores various regulatory options influenced by legal, policy, and economic frameworks to manage and mitigate copyright concerns associated with generative AI and reconcile the interests of IP rights holders with that of generative AI producers. The discussion then introduces techniques to safeguard individual creative works from unauthorized replication, such as watermarking and cryptographic protections. Finally, it describes advanced training strategies designed to prevent AI models from reproducing protected content. In doing so, we highlight key opportunities for action and offer actionable strategies that creators, developers, and policymakers can use in navigating the evolving copyright landscape.
