Large language Models (LLMs) often encounter hallucination issues in knowledge-intensive tasks like question answering (QA). Existing methods attempt to mitigate this by retrieving relevant external knowledge to guide the LLM’s reasoning process. However, knowledge retrieved by semantic similarity-based knowledge retriever is often fragmented and unable to provide structured and coherent reasoning clues that align with the question’s semantic structure. Furthermore, for complex, multi-hop questions involving multiple-topic entities, knowledge retriever frequently introduce irrelevant information or overlook critical knowledge, leading to lower retrieval accuracy for the golden answer and ultimately misguiding the reasoning of LLMs. To address these challenges, we propose a novel model, CoT-F, which introduces responsible, explicit, knowledge-based Chain-of-Thought Families to enhance LLM reasoning. CoT-F identifies high-confidence candidate answers by starting from each topic entity and systematically organizing intermediary triples into structured reasoning paths. Extensive experiments demonstrate that our model significantly improves the reasoning capabilities of LLMs in knowledge-intensive tasks, particularly for complex questions, without requiring additional fine-tuning of the LLM.

Although large language models (LLMs) have shown remarkable capability in using textual documents, leveraging structured knowledge remains a significant challenge. To limit the search scope, recent retrieve-then-read methods have to prune the knowledge graph (KG) prematurely, resulting in the destructive loss of long, multi-hop paths. Although existing semantic parsing methods increase the coverage of answers by executing queries on KGs, similar yet irrelevant noise paths ultimately confuse the LLM, making it difficult to distinguish the correct answer. To address these challenges, we propose a novel backward chaining method, BackChainer, that generates knowledge paths for LLMs by working backward from reasoning possible answers over KGs. We introduce a global KG reasoner to locate candidate answers on the global KG and a constrained path generator to backtrack focused knowledge paths for LLMs. Extensive experiments show that BackChainer outperforms existing KGQA baselines and achieves stable improvements on different LLM backbones. BackChainer demonstrates a high answer hit rate and can integrate with various LLMs using only 0.04

Small language models (SLMs) run quickly, consume little memory, and can be deployed on edge devices, making them especially appealing when compute or energy is limited. Because of these advantages, boosting SLMs' reasoning ability has become an important research goal. A common approach is to distill the long chains of thought (long-CoTs) produced by large reasoning models (LRMs) into SLMs, hoping to transfer the larger models’ strong reasoning ability. However, SLMs do not always benefit from distillation of long-CoTs. The lengthy and complex semantic steps and large amount of self-reflection content in long-CoTs may exceed the limited learning capabilities of SLMs, and the impact of self-reflection density on the performance of SLMs is unclear. To resolve this capacity mismatch, we propose MACoT, a multi-agent framework that synthesizes chains of thought (CoTs) that are more suitable for small models rather than compressing or pruning existing ones. Through the interactive collaboration among six types of agents, MACoT synthesizes semantically explicit, logically clear CoTs that efficiently activate a small model’s internal knowledge through a carefully designed output pattern. At the same time, the CoTs synthesized by our method can retain a small amount of self-reflection content, thereby matching the learning capability of the small model and maximizing its reasoning accuracy. We fine-tuned Qwen2.5-7B-Instruct using only 1879 synthetic CoTs, significantly improving its performance on mathematical reasoning tasks and generalizing well, outperforming models trained on 5x more data. Through experiments, we found that a modest level of self-reflection boosts small-model performance, whereas excessive reflection sharply degrades it, which shows that “teaching SLMs to think” hinges on aligning each CoT’s cognitive load with the model’s capacity.

Among the parameter-efficient fine-tuning (PEFT) methods, LoRA has shown significant advantages in reducing the cost of fine-tuning large language models (LLMs), but its limited representational capacity still constrains performance. Recent studies have attempted to incorporate a Mixture-of-Experts (MoE) structure into LoRA to further enhance representational capacity (MoE-LoRA), but has introduced additional computational overhead. This trade-off highlights a key challenge in the advancement of PEFT methods: how to strengthen representational capacity without sacrificing efficiency. To address this challenge, we revisit the LoRA architecture and find that its intermediate representation naturally forms well-separated clusters after splitting. Motivated by this observation, we propose MKLoRA, which partitions the intermediate representation into multiple subspaces for efficient knowledge collaboration, aggregates knowledge through a shared up-projection matrix, and further optimizes learning with an asymmetric learning-rate strategy. MKLoRA improves representational capacity without increasing the number of trainable parameters, consistently surpassing LoRA variants in single-task and multi-task instruction tuning, while reducing computational overhead to just 57

Temporal knowledge graphs (TKGs) are dynamic heterogeneous multirelational graphs whose topological structures evolve over time. Reasoning over TKGs involves forecasting future events based on complex historical structural information. Efficiently modeling TKGs' structural interactions at each timestamp and the evolution timelines is crucial for later reasoning. However, previous approaches simply learned shallow correlations over TKGs but ignored the heterogeneous deep hierarchy for reasoning. Furthermore, previous evolution modeling techniques for TKGs merely capture the general trends of a given chronological sequence, forgetting the fine-grained event distributions in history. To this end, we propose H-2-Evo, which learns deep Hierarchy of intra-time Heterogeneous interactions and hyperbolic Evolution of inter-time hierarchical dependencies. Specifically, to learn the heterogeneous deep hierarchy in TKGs, we design a heterogeneous hyperbolic message-passing framework with inter-hierarchy perception, intra-hierarchy correlation, and multilayer propagation. It effectively captures the heterogeneous and hierarchical structures in TKGs. To properly capture the temporal evolution of hierarchical structures in TKGs, we devise a novel hyperbolic dual-attentive mechanism, which jointly learns the fine-grained time-varying evolution patterns of TKGs' hierarchical structure and its period-varying evolving trends for later reasoning. Extensive experiments on four public benchmarks demonstrate that H-2-Evo has achieved state-of-the-art reasoning performance.

Supervised fine-tuning (SFT) is central to aligning large language models (LLMs) with instruction following and task-specific reasoning. Despite its success, SFT optimizes token-level likelihoods under the implicit assumption that strictly fitting all tokens in expert demonstrations induces the desired downstream behavior. However, in reasoning tasks where correctness is defined by logical validity or final outcomes rather than exact token realizations, this assumption can lead to optimization misalignment. We empirically observe that low-probability tokens in reasoning demonstrations often correspond to realization-specific or stylistic variations, and that reducing their influence during training consistently improves generalization on reasoning benchmarks. Motivated by this insight, we propose the (BLL-Loss), a simple and parameter-free alternative to standard likelihood training that bounds gradient contributions from low-probability tokens while preserving conventional optimization behavior. We provide theoretical insights and extensive empirical results demonstrating that BLL-Loss improves reasoning generalization across diverse model scales and challenging benchmarks.

Event forecasting is an important task in temporal knowledge graphs (TKGs), aiming to leverage historical memories like humans to make informed decisions about future unknown events. However, according to the multiple intelligence and encoding-retrieval specificity theories, the previous approaches conflict with two fundamental human decision-making paradigms, resulting in incomplete representations of historical events over TKGs and thereby hindering later future forecasting performance. On the one hand, humans recall memories from multiple perspectives, while previous event forecasting works have unilaterally viewed historical sequences as a single-perspective paradigm, resulting in incomplete learning of structural features. On the other hand, the human memory recollection (recall) is a bidirectional process. Existing approaches merely focus on capturing unidirectional evolutionary patterns in chronological order, failing to properly model the time-varying retracing-retrieval processes of memory recollection over time. In this paper, we propose a novel event forecasting method in TKGs, namely Cog-RMH, mimicking the human Cognitive paradigm and Recalling Multiview History to support future decision making. To address the former challenge, we extract inherent structural features of historical concurrent events derived from associative thinking (amygdala), spatial context (hippocampus), and logical reasoning (prefrontal cortex) intelligences, and synthesize their effects on future events. To tackle the latter issue, we propose an encoder-decoder architecture with stacked gated recurrent units that simulate bidirectional memory retracing and retrieval containing cognitive dependencies. We further introduce a retracing-retrieval attention mechanism to model the time-varying human emphasis on different timestamp events during recall. Extensive experiments show that Cog-RMH achieves significantly improved event forecasting performance on four public TKG benchmarks in comparison with the existing state-of-the-art baselines.

Graph anomaly detection (GAD) serves to distinct unusual patterns markedly differ from the norm. Recently, the integration of graph neural networks (GNNs) and contrastive learning for GAD has attracted significant attention. Graph contrastive anomaly detection (GCAD) techniques have primarily emphasized enhancing detection capabilities by integrating multiscale comparison modules. However, as the general backbone of GCAD methods, GNNs derive node representations by smoothing signals from neighboring nodes, which may lead to indistinguishable representations for anomalous nodes. Furthermore, detection outcomes often lack stability due to the inherent randomness of the GCAD pipeline. In our study, we tackle these challenges by proposing a novel approach BWMGNN-GCAD, termed Beta Wavelet Kernel GNN (BWGNN), coupled with a Multi-GNN module. BWGNN mitigates the low-pass issue inherent in traditional GNNs, while employing multiple GNNs enhances result stability. Experiments performed on three popular datasets demonstrate that our method outperforms current state-of-the-art techniques. Ablation studies further confirm that the BWMGNN-GCAD methodology, leveraging multiple GNN modules, significantly boosts detection efficacy. Our research underscores the potential efficacy of employing multiple neural networks and graph augmentation techniques in anomaly detection tasks.

Accurate representation of multimodal knowledge is crucial for event forecasting in real-world scenarios. However, existing studies have largely focused on static settings, overlooking the dynamic acquisition and fusion of multimodal knowledge. 1) At the knowledge acquisition level, how to learn time-sensitive information of different modalities, especially the dynamic structural modality. Existing dynamic learning methods are often limited to shallow structures across heterogeneous spaces or simple unispaces, making it difficult to capture deep relation-aware geometric features. 2) At the knowledge fusion level, how to learn evolving multimodal fusion features. Existing knowledge fusion methods based on static coattention struggle to capture the varying historical contributions of different modalities to future events. To this end, we propose DyMRL, a Dynamic Multispace Representation Learning approach to efficiently acquire and fuse multimodal temporal knowledge. 1) For the former issue, DyMRL integrates time-specific structural features from Euclidean, hyperbolic, and complex spaces into a relational message-passing framework to learn deep representations, reflecting human intelligences in associative thinking, high-order abstracting, and logical reasoning. Pretrained models endow DyMRL with time-sensitive visual and linguistic intelligences. 2) For the latter concern, DyMRL incorporates advanced dual fusion-evolution attention mechanisms that assign dynamic learning emphases equally to different modalities at different timestamps in a symmetric manner. To evaluate DyMRL's event forecasting performance through leveraging its learned multimodal temporal knowledge in history, we construct four multimodal temporal knowledge graph benchmarks. Extensive experiments demonstrate that DyMRL outperforms state-of-the-art dynamic unimodal and static multimodal baseline methods.

Supervised fine-tuning (SFT) is central to aligning large language models (LLMs) with instruction following and task-specific reasoning. Despite its success, SFT optimizes token-level likelihoods under the implicit assumption that strictly fitting all tokens in expert demonstrations induces the desired downstream behavior. However, in reasoning tasks where correctness is defined by logical validity or final outcomes rather than exact token realizations, this assumption can lead to optimization misalignment. We empirically observe that low-probability tokens in reasoning demonstrations often correspond to realization-specific or stylistic variations, and that reducing their influence during training consistently improves generalization on reasoning benchmarks. Motivated by this insight, we propose the (BLL-Loss), a simple and parameter-free alternative to standard likelihood training that bounds gradient contributions from low-probability tokens while preserving conventional optimization behavior. We provide theoretical insights and extensive empirical results demonstrating that BLL-Loss improves reasoning generalization across diverse model scales and challenging benchmarks.

A knowledge graph (KG) is a form of representing knowledge of the objective world. With the expansion of knowledge, KGs frequently incorporate new entities, which often possess limited associated data, known as few-shot features. Addressing the missing knowledge for these emerging entities is crucial practically, but there are significant challenges due to data scarcity. Previously developed methods based on knowledge graph embedding (KGE) and graph neural networks (GNNs) focusing on instance-level KGs are confronted with challenges of data scarcity and model simplicity, rendering them inapplicable to reasoning tasks in few-shot scenarios. To tackle these issues, we propose a multi-view few-shot KG reasoning method for emerging entities. The primary focus of our method lies in resolving the problem of link prediction for emerging entities with limited associated triples from multiple perspectives. Distinct from previous methods, our approach initially abstracts a concept-view KG from the conventional instance-view KG, enabling the formulation of commonsense rules. Additionally, we employ the aggregation of multi-hop subgraph features to enhance the representation of emerging entities. Furthermore, we introduce a more efficient cross-domain negative sampling strategy and a multi-view triple scoring function based on commonsense rules. Our experimental evaluations highlight the effectiveness of our method in few-shot contexts, demonstrating its robustness and adaptability in both cross-shot and zero-shot scenarios, significantly outperforming existing models in these challenging settings.

Large Language Models (LLMs) have demonstrated a remarkable understanding of language nuances through instruction tuning, enabling them to effectively tackle various natural language processing tasks. Recent research has focused on the quality of instruction data rather than the quantity of instructions. However, existing high-quality instruction selection methods rely on external models or rules, overlooking the intrinsic association between pre-trained model and instruction data, making it difficult to select data that align with the preferences of pre-trained model. To address this challenge, we propose a strategy that utilizes noise injection to identify the quality of instruction data, without relying on external model. We also implement the strategy of combining inter-class diversity and intra-class diversity to improve model performance. The experimental results demonstrate that our method significantly outperforms the model trained on the entire dataset and established baselines. Our study provides a new perspective on noise injection in the field of instruction tuning, and also illustrates that the pre-trained model itself should be considered in defining high-quality. Additionally, we publish our selected high-quality instruction data.

Temporal evolution attribute graph prediction, a key task in graph machine learning, aims to forecast the dynamic evolution of node attributes over time. While recent advances in Large Language Models (LLMs) have enabled their use in enhancing node representations for integration with Graph Neural Networks (GNNs), their potential to directly perform GNN-like aggregation and interaction remains underexplored. Furthermore, traditional approaches to initializing attribute embeddings often disregard structural semantics, limiting the provision of rich prior knowledge to GNNs. Current methods also primarily focus on 1-hop neighborhood aggregation, lacking the capability to capture complex structural interactions. To address these limitations, we propose a novel prediction framework that integrates structural information into attribute embeddings through the introduction of an attribute embedding loss. We design specialized prompts to enable LLMs to perform GNN-like aggregation and incorporate a relation-aware Graph Convolutional Network to effectively capture long-range and complex structural dependencies. Extensive experiments on multiple real-world datasets validate the effectiveness of our approach, demonstrating significant improvements in predictive performance over existing methods.

Knowledge graphs are dynamic structures that continuously evolve as new entities emerge, often accompanied by only a handful of associated triples. Current knowledge graph reasoning methods struggle in these few-shot scenarios due to their reliance on extensive structural information.To address this limitation, we introduce ENGRAM, a novel approach that enables inductive reasoning on few-shot KGs by innovatively enriching the semantics from both textual and structural perspectives. Our key innovation lies in designing a task-aware language model that activates the language model’s in-context learning ability for structured KG tasks, effectively bridging the gap between unstructured natural language and structured tasks. Unlike prior methods that inefficiently employ classification over exhaustive candidate sets, we recast knowledge graph reasoning from a generative perspective, allowing for direct computation of inference results without iterative enumeration. Additionally, we propose a distant neighborhood awareness strategy to enrich the sparse structural features of few-shot entities.Our experimental findings indicate that our method not only achieves state-of-the-art performance in few-shot scenarios. The tunable parameters of our model are approximately 1% of those in previous language model-based methods, and the inference time has been reduced to 1/10 of that required by previous methods.

The knowledge graph, a networked structure designed to organize the vast and heterogeneous knowledge existing in the real world, has gained widespread adoption as a background knowledge base for intelligent systems. Nevertheless, the incompleteness of knowledge graphs has been widely recognized as a significant challenge in their development and application. Recently, multi-hop knowledge graph reasoning has been an attractive method for completing a knowledge graph. The multi-hop knowledge graph reasoning based on the reinforcement learning (RL) framework has achieved promising performance in terms of interpretability and scalability. An RL agent automatically reasons over a KG under the guidance of a policy network. When faced with a query, obtaining the approximate range of the answer first and then delving into individual options is a more efficient approach compared to traversing all candidate answers. However, existing RL-based methods have two limitations. First, they lack the ability to filter candidate decisions, making it challenging to handle entities with a large number of neighbors. Second, they fail to consider the intrinsic correlations between entities. To address these limitations, we propose a novel hierarchical knowledge graph reasoning approach HiKGR , which leverages the concept information of entities. Specifically, HiKGR reconstructs the previous action space in RL into a concept space and an instance space, enabling two policies to alternate reasoning at the concept level and the instance level. Furthermore, we propose hierarchical reward functions for the two-level policies to achieve joint optimization. The hierarchical reasoning approach we propose is capable of selecting more reasonable candidate decisions and optimizing the decision space. Experimental results reveal that HiKGR significantly outperforms existing RL-based methods and drastically reduces the action space size.

Augmenting Language Models (LMs) with structured knowledge graphs (KGs) aims to leverage structured world knowledge to enhance the capability of LMs to complete knowledge-intensive tasks. However, existing methods are unable to effectively utilize the structured knowledge in a KG due to their inability to capture the rich relational semantics of knowledge triplets. Moreover, the modality gap between natural language text and KGs has become a challenging obstacle when aligning and fusing cross-modal information. To address these challenges, we propose a novel knowledge-augmented question answering (QA) model, namely, Graph Reasoning Transformers (GRT). Different from conventional node-level methods, the GRT serves knowledge triplets as atomic knowledge and utilize a triplet-level graph encoder to capture triplet-level graph features. Furthermore, to alleviate the negative effect of the modality gap on joint reasoning, we propose a representation alignment pretraining to align the cross-modal representations and introduce a cross-modal information fusion module with attention bias to enable fine-grained information fusion. Extensive experiments conducted on three knowledge-intensive QA benchmarks show that the GRT outperforms the state-of-the-art KG-augmented QA systems, demonstrating the effectiveness and adaptation of our proposed model.