The combination of Mixture-of-Experts (MoE) and Low-Rank Adaptation (LoRA) has shown significant potential for enhancing the multi-task learning capabilities of Large Language Models. However, existing methods face two primary challenges: (1)Imprecise Routing in the current MoE-LoRA method fails to explicitly match input semantics with expert capabilities, leading to weak expert specialization. (2)Uniform weight fusion strategies struggle to provide adaptive update strengths, overlooking the varying complexity of different tasks. To address these limitations, we propose SAMoRA (Semantic-Aware Mixture of LoRA Experts), a novel parameter-efficient fine-tuning framework tailored for task-adaptive learning. Specifically, A Semantic-Aware Router is proposed to explicitly align textual semantics with the most suitable experts for precise routing. A Task-Adaptive Scaling mechanism is designed to regulate expert contributions based on specific task requirements dynamically. In addition, a novel regularization objective is proposed to jointly promote expert specialization and effective scaling. Extensive experiments on multiple multi-task benchmarks demonstrate that SAMoRA significantly outperforms the state-of-the-art methods and holds excellent task generalization capabilities. Code is available at https://github.com/boyan-code/SAMoRA

Job Shop Scheduling Problem (JSSP) is a fundamental combinatorial optimization problem widely applied in real-world scenarios. Building on JSSP, the Dynamic Flexible Job Shop Scheduling Problem (DFJSSP) was introduced to address uncertainties in practical environments. However, existing methods for DFJSSP face notable limitations: (1) they often address only a single type of dynamic factor, limiting their adaptability to complex environments; (2) they still rely on manually designed priority dispatching rules, which lacks generalizability. In this paper, we propose a more challenging Multi-faceted Dynamic Flexible Job Shop Scheduling Problem (MDFJSSP) that simultaneously incorporates multiple dynamic factors. To solve MDFJSSP, we develop a novel Heterogeneous Graph Transformer (HGT) that models the problem as a Dynamic Heterogeneous Graph (DHG) and integrates it with deep reinforcement learning. The HGT is able to effectively learn complex relationships between machines and operations, eliminating the need for priority dispatching rules while maintaining scalability across problem sizes. Experimental results demonstrate that our approach significantly outperforms traditional priority dispatching rules while maintaining strong scalability across problems of varying sizes. Compared with other graph neural networks, our HGT also achieves the state-of-the-art performance.

Temporal Knowledge Graph (TKG) extrapolation aims to predict future events based on historical facts. Recent studies have attempted to enhance TKG extrapolation by integrating TKG's evolving structural representations and textual event chains into Large Language Models (LLMs). Yet, two main challenges limit these approaches: (1) The loss of essential spatial-temporal information due to shallow alignment between TKG's graph evolving structural representation and the LLM's semantic space, and (2) the progressive dilution of the TKG's evolving structural features during LLM fine-tuning. To address these challenges, we propose the Spatial-Temporal Knowledge Adapter (STK-Adapter), which flexibly integrates the evolving graph encoder and the LLM to facilitate TKG reasoning. In STK-Adapter, a Spatial-Temporal MoE is designed to capture spatial structures and temporal patterns inherent in TKGs. An Event-Aware MoE is employed to model intricate temporal semantics dependencies within event chains. In addition, a Cross-Modality Alignment MoE is proposed to facilitate deep cross-modality alignment by TKG-guided attention experts. Extensive experiments on benchmark datasets demonstrate that STK-Adapter significantly outperforms state-of-the-art methods and exhibits strong generalization capabilities in cross-dataset task. The code is available at https://github.com/Zhaoshuyuan0246/STK-Adapter.

Accurate prediction of traffic accident risk at both the road segment and taxi zone granularities is essential for modern intelligent transportation systems. Such predictions offer early warnings to travelers and transportation authorities, contributing to safer mobility. However, achieving accurate traffic accident risk prediction faces two major challenges. Firstly, traffic accident risk is affected by multi-source, entangled and dynamic factors. Accurate accident risk prediction requires a deep under standing of the interactions among these factors, while existing methods struggle to address this task effectively. Secondly, the sparsity of traffic accident data leads to the zero-inflation issue, causing models to appear effective on risk regression metrics while failing to accurately identify high-risk regions. To address these challenges, we propose STKGRisk, a novel model that simultaneously predicts traffic accident risk for both road segments and taxi zones. Specifically, we construct the first spatial temporal knowledge graph for traffic accident risk analysis and design a diachronic embedding module to capture the high-order, dynamic interactions between multi-source factors and traffic accidents. Building upon these embeddings, we develop a spatial temporal graph network encoder to capture the spatial-temporal correlations of accident risk for two granularities from multi-level and multi-view perspectives. To tackle the zero-inflation issue, we design a Zero-Inflated Mixture Poisson decoder to learn the occurrence patterns of sparse accident risk data. Extensive experiments on three real-world traffic accident datasets demonstrate that STKGRisk outperforms existing models. STKGRisk almost achieves the best predictive performance on both the segment and zone-granularity accident risk prediction tasks, and excels particularly in the identification of high-risk areas.

X bot detection aims to automatically identify malicious X bots on the X platform, playing a crucial role in protecting information and maintaining platform stability.Recently, mixture-based methods primarily simultaneously consider investigating various social features (e.g. user metadata, tweets, and social relationships) of users to differentiate humans and bots, which hold excellent performance. However, two major challenges have not been adequately addressed in current mixture-based methods: (1) Humans and bots exhibit different temporal behavior patterns, which has not been fully explored.(2) Existing mixture-based methods promote the detection by fusing diverse features but overlook the noise accumulation that arises during the fusion process.In this paper, we propose a novel X bot detection method with Multiple Social-Temporal Behavior Contrast (MSTBC), which integrates users' multiple social-temporal behaviors, including the static behavior (description content), social behavior (social structure) and temporal behavior (temporal behavior patterns).Specifically, the fine-grained temporal behaviors of users are represented as four different prompts. A temporal behavior PLM with temporal behavior prompts in MSTBC serves as the encoder to understand temporal behavior patterns.In addition, we employ multi-behavior contrast to minimize the differences of various features of users, alleviating the noise accumulation that arises during the fusion of diverse features.Experimental results demonstrate that MSTBC outperforms state-of-the-art models on four datasets. The code is available at https://anonymous.4open.science/r/MSTBC-C659.

Temporal Knowledge Graph (TKG) reasoning has attracted significant attention for completing missing knowledge over time. Recent graph neural network (GNN)-based approaches that explore the temporal evolution of graph topological structures from either continuous-time or discrete-time, which offer distinct perspectives on modeling event associations in TKG. Two GNN-based approaches with different perspectives are supposed to be complementary, but effective integration has not been thoroughly explored in existing research. In addition, capturing the repetitive nature of events during GNN message passing poses a challenge in the continuous-time view, while the complex associations among co-occurring events in KG snapshots cannot be efficiently modeled in the discrete-time view. In this paper, we propose a new Dual-view TKG reasoning network, namely DV-TKR, which comprehensively models the temporal semantic information by integrating the strengths of both types of graph structure encoding representation for reasoning. In DV-TKR, we decompose the quadruple neighbors of each entity into triples and times in the continuous-time TKG. A time-aware event recurring modeling (TERM) module incorporating multiple attention mechanisms in the continuous-time view, is proposed to effectively distinguish the importance of the same triple at different times. For the discrete-time view, we propose a relation-aware graph evolving modeling (RGEM) module to learn the temporal evolution of entities among successive KG snapshots. The relation-aware graph attention mechanism in the RGEM module captures significant correlations among co-occurring events within the overall KG snapshot. Extensive experimental results on three public datasets demonstrate the superiority of our proposed model compared to the state-of-the-art baselines.

Next POI (Point-of-Interest) recommendation aims to forecast users' future movements based on their historical check-in trajectories, holding significant value in location-based services. Existing methods address trajectory data sparsity by integrating rich auxiliary information or using spatial-temporal knowledge graphs (STKGs), showing promising results. Yet, they face two main challenges: i) Due to the difficulty of transforming structured trajectory data into trajectory text describing users' spatial-temporal mobility, the powerful reasoning ability of pre-trained language models is rarely explored to enhance recommendation performance. ii) Methods based on STKG can introduce external knowledge inconsistent with user preferences, leading to the knowledge noise generated hampering the accuracy of recommendations. To this end, we propose a novel approach called STKG-PLM that integrates STKG contrastive learning and prompt pre-trained language model (PLM) to enhance the next POI recommendation. Specifically, we design a spatial-temporal trajectory prompt template that transforms structured trajectories into text corpus based on STKG, serving as the input of PLM to understand the movement pattern of users from coarse-grained and fine-grained perspectives. Additionally, we propose an STKG contrastive learning framework to mitigate the introduced knowledge noise. Extensive experiments on three real-world datasets demonstrate that STKG-PLM exhibits notable performance improvements over the state-of-the-art baseline methods.

Recent studies have combined Mixture of Experts (MoE) and Parameter-Efficient Fine-tuning (PEFT) to fine-tune large language models (LLMs), holding excellent performance in multi-task scenarios while remaining resource-efficient. However, existing MoE approaches still exhibit the following limitations: (1) Current methods fail to consider that different LLM layers capture features at varying levels of granularity, leading to suboptimal performance. (2) Task-level routing methods lack generalizability to unseen tasks. (3) The uncertainty introduced by load imbalance loss undermines the effective specialization of the experts. To address these challenges, we propose HMoRA, a Hierarchical fine-tuning method that combines MoE and LoRA, employing hybrid routing that integrates token-level and task-level routing in a hierarchical manner. This hierarchical hybrid routing allows the model to more efficiently capture both fine-grained token information and broader task contexts. To improve the certainty of expert selection, a novel routing auxiliary loss is introduced. This auxiliary function also enhances the task router's ability to differentiate tasks and its generalization to unseen tasks. Additionally, several optional lightweight designs have been proposed to significantly reduce both the number of trainable parameters and computational costs. Experimental results demonstrate that HMoRA outperforms full fine-tuning across multiple NLP benchmarks, while fine-tuning only 3.9\% of the parameters. The code is available on: https://github.com/LiaoMengqi/HMoRA.

The takeaway recommendation system aims to recommend users' future takeaway purchases based on their historical purchase behaviors, thereby improving user satisfaction and boosting merchant sales. Existing methods focus on incorporating auxiliary information or leveraging knowledge graphs to alleviate the sparsity issue of user purchase sequences. However, two main challenges limit the performance of these approaches: (1) capturing dynamic user preferences on complex geospatial information and (2) efficiently integrating spatial-temporal knowledge from both graphs and sequence data with low computational costs. In this paper, we propose a novel spatial-temporal knowledge distillation model for takeaway recommendation (STKDRec) based on the two-stage training process. Specifically, during the first pre-training stage, a spatial-temporal knowledge graph (STKG) encoder is trained to extract high-order spatial-temporal dependencies and collaborative associations from the STKG. During the second spatial-temporal knowledge distillation (STKD) stage, a spatial-temporal Transformer (ST-Transformer) is employed to comprehensively model dynamic user preferences on various types of fine-grained geospatial information from a sequential perspective. Furthermore, the STKD strategy is introduced to transfer graph-based spatial-temporal knowledge to the ST-Transformer, facilitating the adaptive fusion of rich knowledge derived from both the STKG and sequence data while reducing computational overhead. Extensive experiments on three real-world datasets show that STKDRec significantly outperforms the state-of-the-art baselines.

Recent Continual Learning (CL)-based Temporal Knowledge Graph Reasoning (TKGR) methods focus on significantly reducing computational cost and mitigating catastrophic forgetting caused by fine-tuning models with new data. However, existing CL-based TKGR methods still face two key limitations: (1) They usually one-sidedly reorganize individual historical facts, while overlooking the historical context essential for accurately understanding the historical semantics of these facts; (2) They preserve historical knowledge by simply replaying historical facts, while ignoring the potential conflicts between historical and emerging facts. In this paper, we propose a Deep Generative Adaptive Replay (DGAR) method, which can generate and adaptively replay historical entity distribution representations from the whole historical context. To address the first challenge, historical context prompts as sampling units are built to preserve the whole historical context information. To overcome the second challenge, a pre-trained diffusion model is adopted to generate the historical distribution. During the generation process, the common features between the historical and current distributions are enhanced under the guidance of the TKGR model. In addition, a layer-by-layer adaptive replay mechanism is designed to effectively integrate historical and current distributions. Experimental results demonstrate that DGAR significantly outperforms baselines in reasoning and mitigating forgetting.

The emergence of Graph Neural Networks (GNNs) has greatly advanced the development of recommendation systems. Recently, many researchers have leveraged GNN-based models to learn fair representations for users and items. However, current GNN-based models suffer from biased user–item interaction data, which negatively impacts recommendation fairness. Although there have been several studies employing adversarial learning to mitigate this issue in recommendation systems, they mostly focus on modifying the model training approach with fairness regularization and neglect direct intervention of biased interaction. In contrast to these models, this article introduces a novel perspective by directly intervening in observed interactions to generate a counterfactual graph (called FairGap) that is not influenced by sensitive node attributes, enabling us to learn fair representations for users and items easily. We design FairGap to answer the key counterfactual question: “Would interactions with an item remain unchanged if a user’s sensitive attributes were concealed?”. We also provide theoretical proofs to show that our learning strategy via the counterfactual graph is unbiased in expectation. Moreover, we propose a fairness-enhancing mechanism to continuously improve user fairness in the graph-based recommendation. Extensive experimental results against state-of-the-art competitors and base models on three real-world datasets validate the effectiveness of our proposed model.

Temporal knowledge graphs (TKGs) have been identified as a promising approach to represent the dynamics of facts along the timeline. The extrapolation of TKG is to predict unknowable facts happening in the future, holding significant practical value across diverse fields. Most extrapolation studies in TKGs focus on modeling global historical fact repeating and cyclic patterns, as well as local historical adjacent fact evolution patterns, showing promising performance in predicting future unknown facts. Yet, existing methods still face two major challenges: (1) They usually neglect the importance of historical information in KG snapshots related to the queries when encoding the local and global historical information; (2) They exhibit weak anti-noise capabilities, which hinders their performance when the inputs are contaminated with noise.To this end, we propose a novel Local-global history-aware Contrastive Learning model (LogCL) for TKG reasoning, which adopts contrastive learning to better guide the fusion of local and global historical information and enhance the ability to resist interference. Specifically, for the first challenge, LogCL proposes an entity-aware attention mechanism applied to the local and global historical facts encoder, which captures the key historical information related to queries. For the latter issue, LogCL designs four historical query contrast patterns, effectively improving the robustness of the model. The experimental results on four benchmark datasets demonstrate that LogCL delivers better and more robust performance than the state-of-the-art baselines.

To alleviate the class imbalance issue in unlabeled data for zero-shot text classification task and fully utilize the reasoning ability of pre-trained language models,this paper proposes a Knowledge Enhanced Zero-shot Text Classification(KE0TC)method.This method uses prompt templates to guide large pre-trained language model to extend category labels and builds a knowledge graph,using the graph structure for denoising and self-supervised data generation.Paragraph sampling and other methods are used to map the extracted labeled data into the parameter space of the classifier,thus achieving modeling of the classification space without collecting unlabeled training data.Compared with four baseline models on three text classification datasets,the proposed method achieves higher classification performance with lower time-consuming and less corpus.

Graph neural networks (GNNs) have shown promising performance for knowledge graph reasoning. A recent variant of GNN called progressive relational graph neural network (PRGNN), utilizes relational rules to infer missing knowledge in relational digraphs and achieves notable results. However, during reasoning with PRGNN, two important properties are often overlooked: (1) the sequentiality of relation composition, where the order of combining different relations affects the semantics of the relational rules, and (2) the lagged entity information propagation, where the transmission speed of required information lags behind the appearance speed of new entities. Ignoring these properties leads to incorrect relational rule learning and decreased reasoning accuracy. To address these issues, we propose a novel knowledge graph reasoning approach, the Relational rUle eNhanced Graph Neural Network (RUN-GNN). Specifically, RUN-GNN employs a query related fusion gate unit to model the sequentiality of relation composition and utilizes a buffering update mechanism to alleviate the negative effect of lagged entity information propagation, resulting in higher-quality relational rule learning. Experimental results on multiple datasets demonstrate the superiority of RUN-GNN is superior on both transductive and inductive link prediction tasks.

In recent years,graph neural networks have shown great potential in knowledge graph link prediction,es-pecially progressive relational graph neural networks.This paper proposed an Entity Context-Aware Progressive Re-lational Graph Attention Network to better understand relational context during reasoning by using Relational Con-text Generation Network and Relational Context Aware Network,and fully exploits the entity's dependent context information through a buffer layer mechanism.Experiments on multiple knowledge graph datasets show excellent performance of the proposed method in both transductive and inductive link prediction tasks.

The devastating, ongoing Covid-19 epidemic has led to many students resorting to online education. In order to better guarantee the quality, online education faces severe challenges. There is an important part of online education referred to as Knowledge Tracing (KT). The objective of KT is to estimate students’ learning performance using a series of questions. It has garnered widespread attention ever since it was proposed. Recently, an increasing number of research efforts have concentrated on deep learning (DL)-based KT attributing to the huge success over traditional Bayesian-based KT methods. Most existing DL-based KT methods utilize Recurrent Neural Network and its variants, i.e. Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU) etc. Recurrent neural networks are good at modeling local features, but underperforms at long sequence modeling, so the attention mechanism is introduced to make up for this shortcoming. In this paper, we introduce a DL-based KT model referred to as Convolutional Attention Knowledge Tracing (CAKT) utilizing attention mechanism to augment Convolutional Neural Network (CNN) in order to enhance the ability of modeling longer range dependencies.

Next Point-of-Interest (POI) recommendation has shown great value for both users and businesses in the field of location-based services. Many spatial–temporal inferring methods have been developed to perform this task, but the data sparsity of POI trajectories greatly hinders the recommendation performance. Knowledge graphs (KGs) have been demonstrated as an effective way to tackle data sparsity in the general recommendation field by leveraging the valuable information of entities and relations. Yet, few studies have explored applying KGs for the next POI recommendation task because of the following challenges: (1) how to represent the dynamic mobility behaviors of users with the static entities and relations in KGs; and (2) how to utilize the different types of entities and relations in KGs to capture long- and short-term preferences of users. In this work, we investigate building a spatial–temporal KG (STKG) from check-in sequences of users to promote the next POI recommendation, without introducing any external attributes of users and POIs. In STKG, we design a novel spatial–temporal transfer relation to intuitively capture users’ transition patterns between neighboring POIs. Then, based on the STKG, we propose an innovative model, named STKGRec, for the next POI recommendation, which explicitly models long- and short-term preferences of users in an end-to-end manner. In particular, STKGRec learns both the spatial–temporal correlation of consecutive and nonconsecutive visits in the current check-in sequence to comprehensively capture the short-term preferences of users. Extensive experiments on four real-world datasets demonstrate the superiority of STKGRec against the state-of-the-art baseline methods. The code of our proposed model is available athttps://github.com/WeiChen3690/STKGRec.

Classic DNN pruning mostly leverages software-based methodologies to tackle the accuracy/speed tradeoff, which involves complicated procedures like critical parameter searching, fine-tuning and sparse training to find the best plan. In this paper, we explore the opportunities of hardware runtime pruning and propose a hardware runtime pruning methodology, termed as "BitX" to empower versatile DNN inference. It targets the abundant useless bits in the parameters, pinpoints and prunes these bits on-the-fly in the proposed BitX accelerator. The versatility of BitX lies in: (1) software effortless; (2) orthogonal to the software-based pruning; and (3) multi-precision support (including both floating point and fixed point). Empirical studies on image classification and object detection models highlight the following results: (1) up to 4.82x speedup over the original non-pruned DNN and 14.76x speedup collaborated with the software-pruned DNN; (2) up to 0.07% and 0.9% higher accuracy for the floating-point and fixed-point DNN, respectively; (3) 2.00x and 3.79x performance improvement over the state-of-theart accelerators, with 0.039mm2 and 68.62 mW (floating-point 32), 36.41 mW(16-bit fixed point) power consumption under TSMC 28 nm technology library.

Deep learning systems have been phenomenally successful in the fields of computer vision, speech recognition, and natural language processing. Recently, researchers have adopted deep learning techniques to tackle collaborative filtering with implicit feedback. However, the existing methods generally profile both users and items directly, while neglecting the similarities between users' and items' neighborhoods. To this end, we propose the neighborhood attentional memory networks (NAMN), a deep learning recommendation model applying two dedicated memory networks to capture users' neighborhood relations and items' neighborhood relations respectively. Specifically, we first design the user neighborhood component and the item neighborhood component based on memory networks and attention mechanisms. Then, by the associative addressing scheme with the user and item memories in the neighborhood components, we capture the complex user-item neighborhood relations. Stacking multiple memory modules together yields deeper architectures exploring higher-order complex user-item neighborhood relations. Finally, the output module jointly exploits the user and item neighborhood information with the user and item memories to obtain the ranking score. Extensive experiments on three real-world datasets demonstrate significant improvements of the proposed NAMN method over the state-of-the-art methods.

The efficient and accurate prediction of building energy consumption can improve the management of power systems. In this paper, the rough set theory was used to reduce the redundant influencing factors of building energy consumption and find the critical factors of building energy consumption. These key factors were then used as the input of a deep neural network with a "deep" architecture and powerful capabilities in extracting features. Building energy consumption is output of the deep neural network. This study collected data from 100 civil public buildings for rough set reduction, and then collected data from a laboratory building of a university in Dalian for nearly a year to train and test deep neural networks. The test included both the short-term and medium-term predictions of building energy consumption. The prediction results of the deep neural network were compared with that of the back propagation neural network, Elman neural network and fuzzy neural network. The results show that the integrated rough set and deep neural network was the most accurate. The method proposed in this study could provide a practical and accurate solution for building energy consumption prediction. (c) 2021 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).