Solution solvers rely on sophisticated heuristics honed through decades of research to tackle various instances of combinatorial optimization (CO) encountered in practical scenarios. Recent studies demonstrate that leveraging the data’s shared internal structure of instances allows reinforcement learning (RL) to achieve state-of-the-art performance in mastering advanced heuristics. However, existing RL-based algorithms depend on trial-and-error and extensive reward engineering tailored to individual data, leading to time-consuming and inefficient real-world applications. Moreover, these methods, incorporating construction heuristics, still fall short in accuracy compared to most solution solvers, limiting their practical utility. This paper introduces RAIL*, a novel framework that integrates RL and generative adversarial imitation learning (GAIL) to address the challenge by searching in branch-and-bound (B&B) algorithms. RAIL* adopts a policy architecture with dual decoders, aligning with RL’s sequence decoding and GAIL’s edge decoding. RL provides expert trajectories for GAIL to imitate, while GAIL utilizes the learned reward function to reciprocate to RL. The collaboration between the two components facilitates iterative improvements in the learned policy and reward function by complementing and constraining each other.

With China's commitment to achieving a carbon peak and carbon neutralisation by 2030 and 2060, respectively, the government's requirements for enterprises to adopt green technology are becoming increasingly urgent. We established a tripartite evolutionary game model of an enterprise, government and bank. We analysed conditions for the existence of stable points and factors affecting the choice of stakeholder's strategy. The influence of the change in each parameter on the evolutionary stable strategy was further studied. We found that the government's reward, subsidy and taxes for enterprise's green technology innovation can increase the willingness of enterprises to implement green technology innovation, and the role of taxes is greater than that of subsidies. The preferential interest rate for green innovation is conducive to stimulating enterprises to engage in green innovation, but the increase in the cost of green credit will reduce the willingness of banks to provide green credit. With the increase in the subsidy, incentives and the cost of environmental regulation, the policy burden of the government will gradually increase, thus reducing the willingness of the government to implement environmental regulation policy. The research presented in this paper furthers work on the theoretical modelling of green credit and environmental regulation policy.

Multi-objective optimization (MOO) endeavors to identify optimal solutions from a finite array of possibilities. In recent years, deep reinforcement learning (RL) has exhibited promise through its well-crafted heuristics in tackling NP-hard combinatorial optimization (CO) problems. Nonetheless, current methodologies grapple with two key challenges: (1) They primarily concentrate on single-objective optimization quandaries, rendering them less adaptable to the more prevalent MOO scenarios encountered in real-world applications. (2) These approaches furnish an approximate solution by imbibing heuristics, lacking a systematic means to enhance or substantiate optimality. Given these challenges, this study introduces an overarching hybrid strategy, dynamic programming with meta-reinforcement learning (DPML), to resolve MOO predicaments. The approach melds meta-learning into an RL framework, addressing multiple subproblems inherent to MOO. Furthermore, the precision of solutions is elevated by endowing exact dynamic programming with the prowess of meta-graph neural networks. Empirical results substantiate the supremacy of our methodology over previous RL and heuristics approaches, bridging the chasm between theoretical underpinnings and real-world applicability within this domain.

Large Language Models (LLMs) have shown impressive performance in various tasks, including knowledge graph completion (KGC). However, current studies mostly apply LLMs to classification tasks, like identifying missing triplets, rather than ranking-based tasks, where the model ranks candidate entities based on plausibility. This focus limits the practical use of LLMs in KGC, as real-world applications prioritize highly plausible triplets. Additionally, while graph paths can help infer the existence of missing triplets and improve completion accuracy, they often contain redundant information. To address these issues, we propose KG-CF, a framework tailored for ranking-based KGC tasks. KG-CF leverages LLMs' reasoning abilities to filter out irrelevant contexts, achieving superior results on real-world datasets. The code and datasets are available at .

Large Language Models (LLMs) represent an advanced evolution of earlier, simpler language models. They boast enhanced abilities to handle complex language patterns and generate coherent text, images, audios, and videos. Furthermore, they can be fine-tuned for specific tasks. This versatility has led to the proliferation and extensive use of numerous commercialized large models. However, the rapid expansion of LLMs has raised security and ethical concerns within the academic community. This emphasizes the need for ongoing research into security evaluation during their development and deployment. Over the past few years, a substantial body of research has been dedicated to the security evaluation of large-scale models. This article an in-depth review of the most recent advancements in this field, providing a comprehensive analysis of commonly used evaluation metrics, advanced evaluation frameworks, and the routine evaluation processes for LLMs. Furthermore, we also discuss the future directions for advancing the security evaluation of LLMs.

The widespread use of knowledge graphs in various fields has brought about a challenge in effectively integrating and updating information within them. When it comes to incorporating contexts, conventional methods often rely on rules or basic machine learning models, which may not fully grasp the complexity and fluidity of context information. This research suggests an approach based on reinforcement learning (RL), specifically utilizing Deep Q Networks (DQN) to enhance the process of integrating contexts into knowledge graphs. By considering the state of the knowledge graph as environment states defining actions as operations for integrating contexts and using a reward function to gauge the improvement in knowledge graph quality post-integration, this method aims to automatically develop strategies for optimal context integration. Our DQN model utilizes networks as function approximators, continually updating Q values to estimate the action value function, thus enabling effective integration of intricate and dynamic context information. Initial experimental findings show that our RL method outperforms techniques in achieving precise context integration across various standard knowledge graph datasets, highlighting the potential and effectiveness of reinforcement learning in enhancing and managing knowledge graphs.

Vehicle routing optimization is a crucial responsibility of transportation service providers, which can significantly reduce operating expenses and improve client satisfaction. Learning to tackle routing optimization problems automatically can be the next significant step forward in optimization technology. Despite recent advancements in automatically learned heuristics for routing optimization problems, state-of-the-art traditional methods such as Lin-Kernighan-Helsgaun (LKH) still outperform machine learning-based approaches. To narrow this gap, we propose a novel technique called self-supervised reinforcement learning (SSRL), which combines self-supervised learning with the LKH heuristic. We provide a node decoder and an edge decoder corresponding to reinforcement learning and self-supervised learning for learning node penalties and edge scores, respectively. The self-supervised part with cross-entropy loss offers strong gradient signals for parameter updates. At the same time, the reinforcement learning component functions as a regularizer to drive the supervised part, which focuses on particular rewards. SSRL learns and replicates all of the LKH’s significant components, improving the original LKH’s generalization and performance. Through experiments on multiple vehicle routing problems, SSRL has demonstrated superior accuracy and efficiency compared to existing methods. Our results provide empirical evidence of SSRL’s effectiveness and potential as a promising solution for optimizing complex routing problems.

Offloading tasks from edge devices to the cloud is an important method to enhance the performance of the edge device. With the help of EH (Energy Harvesting) technology, the edge device can use the collected green energy to support its operations. Most offloading scheduling methods use as much green energy as the edge device collected. Unlike prior research, we consider the long-term benefits of energy. In this paper, we forecast the solar energy supply with meteorological methods which are based on the weather forecast data. Then, we use quadratic programming to allocate energy based on the forecast energy to maximize energy efficiency. Finally, we use NSGA (Non-dominated Sorting Genetic Algorithms) to offload tasks in the edge device. Simulations show that our proposed method not only minimizes the execution time and the energy consumption of clouds, but also enhances the QoE of users.

Vehicle routing (VRP) and traveling salesman problems (TSP) are classical and interesting NP-hard routing combinatorial optimization (CO) with practical significance. While moving forward with artificial intelligence, researchers are paying more and more attention to applying machine learning to classical CO problems. However, traditional reinforcement learning faces challenges like reward sparsity and unstable training, so it is necessary to assist agents in finding high-quality routings in the initial model training stage to obtain more positive feedback. This paper proposes a novel Monte Carlo Tree Search (MCTS)-based two-stage multi-agent reinforcement learning training pipeline (MCRL) in which we also design a multifunctional reward function, improving efficiency, accuracy, and diversity to guide agents to learn the routings over graphs better. Besides, previous approaches are frequently too sluggish in runtime to be useful in contexts with sparsely connected networks and uncertain traffic. As an alternative, we design a model based on graph neural networks that can execute multi-agent routing in a sparsely connected graph with constantly changing traffic circumstances. Also, the agents are better equipped to collaborate online and adjust to changes thanks to our learned communication module.

With the emergence of the sharing economy and the rapid growth of mobile communications technologies, many novel sharing service models have been developed stemming from ride-hailing. Urban traffic congestion, coupled with energy conservation and emissions reduction, has prompted research on enhancing vehicle seat utilization in taxi service. To offer more effective and reliable ride-hailing, we consider ride-sharing problem with passenger transfer that allows passegers to transfer between vehicles at transfer stations. The problem requires simultaneous addressing the issues of request dispatching, transfer scheduling, and vehicle rebalancing. Studying such a ride-hailing model, we propose a novel joint decision framework combining deep reinforcement learning (DRL) with integer-linear programming (ILP) to solve the problem. We use ILP to obtain the optimal online dispatching and matching strategy in each decision stage, and DRL to learn the approximate state value of each vehicle that incorporates with some strategies to limit the state space and reduce the computational complexity. Performing numerical studies on the real-world trip dataset in Chengdu, we demonstrate that the proposed method outperforms several state-of-the-art methods, and that ride-sharing with passenger transfer is more beneficial than traditional ride-sharing.

Combinatorial optimization, such as vehicle routing and traveling salesman problems for graphs, is NP-hard and has been studied for decades. Many methods have been proposed for its possible solution, including, but not limited to, exact algorithms, approximate algorithms, heuristic algorithms, and solution solvers. However, these methods cannot learn the problem's internal structure nor generalize to similar or larger-scale problems. Recently, deep reinforcement learning has been applied to combinatorial optimization and has achieved convincing results. Nevertheless, the challenge of effective integration and training improvement still exists. In this study, we propose a novel framework (BDRL) that combines BERT (Bidirectional Encoder Representations from Transformers) and deep reinforcement learning to tackle combinatorial optimization over graphs by treating general optimization problems as data points under an identified data distribution. We first improved the transformer encoder of BERT to embed the combinatorial optimization graph effectively. By employing contrastive objectives, we extend BERT-like training to reinforcement learning and acquire self-attention-consistent representations. Next, we used hierarchical reinforcement learning to pre-train our model; that is, to train and fine-tune the model through an iterative process to make it more suitable for a specific combinatorial optimization problem. The results demonstrate our proposed framework's generalization ability, efficiency, and effectiveness in multiple tasks.

Deep reinforcement learning (DRL) has shown promise in solving challenging combinatorial optimization (CO) problems, such as the traveling salesman problem (TSP) and vehicle routing problem (VRP). However, existing DRL methods rely on manually designed reward functions, which may be inaccurate or unrealistic. Moreover, traditional DRL algorithms suffer from unstable training and sparse reward problems. This paper proposes GIRL (Generative Inverse Reinforcement Learning), a method to learn 2-opt heuristics without explicit extrinsic rewards to address these limitations. GIRL combines generative adversarial networks (GANs) and DRL to learn effective policies and reward functions in a reverse end-to-end fashion, improving generalization capabilities. Furthermore, we introduce a self-attentional policy network tailored for 2-opt heuristics and train the framework using a soft actor-critic algorithm along with a discriminator in the GAN. Extensive experiments on various TSP and VRP instances demonstrate superior performance compared to state-of-the-art methods. Moreover, integrating GANs and DRL enables data-driven reward functions, improving accuracy and realism. Using self-attentional networks and the soft actor-critic algorithm enhances training stability and addresses the sparse reward problem. This work advances reinforcement learning techniques in CO, enabling more accurate and practical optimization methods in real-world applications.

Relationship prediction in knowledge graph aims to identify and infer new relationships from existing data, and provides knowledge services for many downstream tasks. At present, many researches solve the link prediction problem between entities by mapping entities and relations into a vector space or searching the paths between entities. These methods only consider the influence of single path or first-order information but ignore more complex relation information between entities. Therefore, this paper proposes a novel link prediction method based on subgraph reasoning in knowledge graph, uses the subgraph structure to obtain the entity pair neighborhood structure information, combines the advantages of representation learning and path reasoning, and realizes the relationship prediction between entities. This paper first extends the paths between entities to subgraphs, constructs node subgraph and relationship subgraph from entity level and relationship level respectively, then combines the graph embedding representation with the graph neural network to calculate the subgraph features, to get richer entity characteristics and relationship characteristics. Finally, this paper calculates the neighborhood structure information of entity pairs from the subgraph structure to conduct link prediction between entities. Experimental results demons-trate that the proposed approach outperforms other reasoning-based link prediction methods on two benchmark datasets.

Learning heuristics without expert experience to construct solutions automatically has always been a critical challenge of combinatorial optimization. It is also the pursuit of artificial intelligence to construct an agent with the planning ability to solve multiple problems simultaneously. Nonetheless, most current learning-based methods for combinatorial optimization still rely on artificially designed heuristics. In real-world problems, the environment’s dynamics are often unknown and complex, making it challenging to generalize and implement current methods. Inspired by AlphaGo Zero, we propose a novel self-play reinforcement learning algorithm (CH-Zero) based on the Monte Carlo tree search (MCTS) for routing optimization problems in this paper. Like AlphaGo Zero, CH-Zero does not require expert experience but some necessary rules. However, unlike other self-play algorithms based on MCTS, we have designed offline training and online reasoning. Specifically, we apply self-play reinforcement learning without MCTS to train offline policy and value networks. Then, we apply the learned heuristics and neural network combined with an MCTS to make inferences on unknown instances. Since we did not incorporate MCTS during training, this is equivalent to training a lightweight self-playing framework whose learning efficiency is much higher than the existing self-play-based methods for combinatorial optimization. We can employ the learned heuristics to guide MCTS to improve policies and take better actions at runtime.

This paper studied a tripartite evolutionary game of stakeholders in environmental pollution control. Most previous studies on this issue are limited to a focus on system dynamics with two-party game problems and lack a spatial analysis of strategy evolution. The parameters adopted are too few, and the influencing factors considered are too simple. The purpose of the paper is to introduce more parameters to study, which will have an important impact on the strategy choices of participants and the evolution path of the strategy over time. We construct a tripartite evolutionary game model of sewage enterprises, governments and the public. We establish a payment matrix and replicator equations as our method, and we also implement parameter simulations in MATLAB. In summary, we found that the reward and punishment mechanism plays an important role in environmental pollution control. Specifically: intensifying rewards and penalties will help encourage sewage enterprises to meet the discharge standard and the public to participate in pollution control action. However, increased rewards will reduce government's willingness to adopt incentive strategies; Government's reward for public's participation in the action must be greater than the increased cost of participation; Reducing the cost of sewage enterprise can also encourage them to implement standard emissions. The research presented in this paper further improves standard emissions and designs reasonable reward and punishment mechanism.

Recent studies have shown that reinforcement learning (RL) can provide state-of-the-art performance at learning sophisticated heuristics by exploiting the shared internal structure combinatorial optimization instances in the data. However, existing RL-based methods require too much trial-and-error reliant on sophisticated reward engineering, which is laborious and inefficient for practical applications. This paper proposes a novel framework (RAIL) that combines RL and generative adversarial imitation learning (GAIL) to meet the challenge by searching in branch-and-bound algorithms. RAIL has a policy architecture with dual decoders, corresponding to the sequence decoding of RL and the edge decoding of GAIL, respectively. The two complement each other and restrict each other to improve the learned policy and reward function iteratively.

The combinatorial optimization (CO) problems on the graph are the core and classic problems in artificial intelligence (AI) and operations research (OR). For example, the Vehicle Routing Problem (VRP) and Traveling Salesman Problem (TSP) are fascinating NP-hard problems and have important significance for the existing transportation system. Traditional methods such as heuristics methods, exact algorithms, and solution solvers can already find approximate solutions on small-scale graphs. However, they are helpless for large-scale graphs and other problems with similar structures. Moreover, traditional methods often require artificially designed heuristic functions to aid decision-making. In recent years, more and more work has focused on applying deep learning and reinforcement learning (RL) to learn heuristics, which allows us to learn the internal structure of the graph end-to-end and find the optimal path under the guidance of heuristic rules. However, most of these still need manual assistance, and the RL method used has the problems of low sampling efficiency and small searchable space. This paper proposes a novel framework (called OmegaZero) based on Alphago Zero, which does not prescribe expert experience or label data but is trained through self-play. We divide the learning into two stages: in the first stage, we employ graph attention network (GAT) and GRU to learn node representations and memory history trajectories. In the second stage, we employ Monte Carlo tree search (MCTS) and deep RL to search for the solution space and train the model.

The vehicle routing problem as a classic NP-hard problem could be optimized by path choices due to its practical application value. This study proposes a novel variational autoencoder framework for path optimization on graphs, involving graph neural networks and generative adversarial networks. We took the center node as the root node to divide the graph into different subgraphs and find the nodes that compose the optimal solution through variational reasoning. We next used reinforcement learning to optimize the entire variational framework end-to-end. This contribution can also apply in both modeling and training combinatorial optimization over graphs. An extensive experiment on different scales of traveling salesman and vehicle routing instances was conducted. The findings indicate that our framework is efficient and effective in learning and reasoning, and its accuracy and generalization outperform the baselines.