In mixed-motive games, agents communicate to cooperate and compete, resulting in dynamic alliances and latent strategic intents. Although recent end-to-end learning-based systems perform well in mixed-motive games, alliance prediction and strategic intents are typically not modeled as explicit reasoning targets, limiting interpretability and structured relational analysis. To address this limitation, we propose MA-SRHG, a data driven graph-based multi-agent strategic reasoning framework that jointly models alliance prediction and strategic intent reasoning in mixed-motive games. MA-SRHG constructs a temporal heterogeneous interaction graph that captures communication and action patterns, and performs hierarchical relational reasoning across individual, local, and global levels, disentangling alliance structures and inferring latent strategic intents from interaction data. Furthermore, MA-SRHG integrates alliance prediction modeling and strategic intent reasoning via a dual closed-loop coordination mechanism with shared representations and feedback-driven refinement. We evaluated MA-SRHG on the human game dataset for two tasks: alliance transition prediction and strategic intent reasoning. Experimental results demonstrate that our framework outperforms sequence-based and graph-based baselines in alliance transition prediction and intent reasoning. More importantly, MA-SRHG provides interpretable relational explanations for alliance transitions and intent changes, offering a structured and interpretable approach to relational reasoning in complex multi-agent interaction settings.

Multi-agent reinforcement learning (MARL) has achieved notable progress in cooperative tasks, yet the common use of parameter sharing often leads to behavioral homogeneity, which restricts the diversity and adaptability of agent coordination. To address this issue, we propose Contrastive Learning for Role Representation (CLRE), a framework designed to promote strategic diversity by encouraging agents to learn distinct and semantically meaningful roles. The key idea is to employ a Large Language Model (LLM) as an offline domain expert that automatically generates semantic role labels from agent states. These labels provide reliable supervision for a contrastive learning objective, guiding the model to acquire role embeddings that are both discriminative and interpretable. To ensure these role representations effectively support coordination, CLRE incorporates a graph attention network that captures inter-agent relationships and a dual-attention value function that integrates role information with the global state for context-aware decision-making. Experiments on the StarCraft Multi-Agent Challenge (SMAC) benchmark demonstrate that CLRE achieves superior win rates and sample efficiency compared with state-of-the-art MARL methods. Ablation studies further confirm that each component contributes to the overall performance, showing that the proposed approach enables diverse and efficient cooperation among agents in complex environments.

The challenge in cooperative multi-agent reinforcement learning (MARL) lies in designing credit assignment mechanisms that incentivize cooperation among agents. While existing methods based on Shapley value provide interpretability, their over-reliance on expected returns introduces critical limitations—specifically, the approach fails to account for environmental risk, which in turn undermines its interpretability in credit assignment under risky conditions. Hence, this paper proposes a \textbf{\underline{ri}}sk-\textbf{\underline{s}}ensitive advantage Shapley value-based \textbf{\underline{c}}redit \textbf{\underline{a}}ssignment method (RISCA), which introduces a novel integration of advantage functions with distributed RL frameworks to achieve risk-sensitive credit assignment. Specifically, RISCA consists of three core components: (1) Constructing the Advantage Markov Shapley Value (AMSV), which integrates advantage functions into marginal contribution calculations to capture the risk impacts of individual actions; (2) Quantile regression for learning action value distributions, dynamically optimizing risk-sensitive credit assignment via Conditional Value at Risk; (3) Risk-sensitive joint policy learning under the centralized training with decentralized execution framework. We evaluate RISCA in three benchmarks under risk conditions. Experimental results demonstrate that RISCA outperforms existing MARL algorithms, not only enhancing coordination performance but also improving the interpretability under risk conditions.

In multi-agent reinforcement learning (MARL), a critical challenge arises when self-interested agents possess asymmetric information: a more-informed agent (sender) may send distorted information to seek more personal gain, while a less-informed agent (receiver) can only passively receive this information and take actions, resulting in suboptimal social welfare. To bridge this gap, this paper presents a model of Markov Incentive-Signaling Game (MISG), in which the receiver may dynamically design and release an incentive mechanism to motivate the sender to truthfully disclose her private information. Moreover, we propose a tri-level optimization method to jointly optimize the sender's signaling scheme, the receiver's incentive scheme, and the receiver's action policy. In particular, we design the mechanism gradient to handle non-stationarity caused by concurrent policy adaptations, and employ constraint optimization to guarantee incentive compatibility of the mechanism. Experiments demonstrate that our approach achieves Pareto improvements in both individual payoffs and social welfare. Our work demonstrates that incentive mechanisms effectively address the goal misalignment challenge in MARL systems with information asymmetry.

In multi-task generalization, multi-agent reinforcement learning (MARL) often faces the challenge of differences in the number of agents between the source and target tasks. Current multi-task MARL methods attempt to learn cooperation skills as the basis for generalization, but these skills are quite sensitive to offline data. This paper finds that agents adopting similar risk responses when facing potential risks in different tasks contributes to enhancing the generalization of their cooperation policy. Specifically, this paper proposes an environmental risk knowledge (RIKD) representation and response method to achieve generalizable multi-agent cooperation policy learning in risk conditions. RIKD first trains a risk attitude selector based on offline data to learn the mapping from risk knowledge to risk attitudes. Then, RIKD coordinates the actions of agents with different risk attitudes through a mixing network, and learns a risk knowledge extraction network for agents that can extract risk knowledge from the current observation.Additionally, to enhance RIKD's ability to adapt to environmental risk changes in new tasks, we have provided its online version, RIKD (online). Empirical results on MARL benchmarks demonstrate that RIKD significantly improves the multitask generalization performance of cooperative policies under risk conditions.

Multi-domain competition is developing for disintegrating the component of the opponent's operational system and winning advantage in decision space. Island air defense is a typical multi-domain security problem, which dramatically increases the complexity of decision-making by considering different factors such as multi-stages decisions, multi-domain settings, imperfection information, and uncertain events. However, current research on island air defense security problems is sparse and lacks consideration of key factors. To provide support for assisting human commanders to take wise decisions in a complex environment, we build a multi-domain multi-state island air defense model and propose responding solving algorithms. We study the whole progress of island air defense and propose a multi-domain, multi-stage imperfection information security game that formulates critical characters in the adversarial scenario of island air defense. In addition, considering a bounded rational opponent's possible strategies, we propose an opponent-aware Monte Carlo counterfactual regret minimization algorithm for learning a robust defensive strategy in the security game. We evaluate our methods in various adversarial scenarios. The results show that our equilibrium learning method can effectively play against an opponent with bounded rationality and significantly outperform some advanced algorithms.

In Imperfect Information Games (IIGs) with large state spaces, recognizing the opponent's policy and adopting a targeted policy are typical strategies in opponent modeling for exploiting an opponent. However, quickly and accurately recognize the opponent's policy in online matches remains a challenge due to the limited observed data.This paper proposes a novel opponent recognition method based on Game History Tree (GHT). The GHT is designed to transform limited game histories into a tree-like graph, and is fed into a graph classifier based on Graph Neural Network (GNN) to enable end-to-end recognition.Experimental results on Texas Hold'em show that this method achieves higher accuracy and quicker recognition compared to baseline methods. It reduces the average number of games required to recognize the opponent's static and dynamic policies by at least 56.5% and 54.2%, respectively, compared to the best baseline method. These results highlight that the GHT serves as an effective feature engineering model for policy representation and provides a new approach to identify the individual policy of the opponent in two-player large state space IIGs.

Traditional reinforcement learning (RL) methods typically assume complete observation data, but real-world scenarios are often far more complex. Real-world deployments face missing data where part of the state dimensions become unavailable due to sensor failures, communication blackouts, or temporal sampling discontinuities. While recent research has achieved significant progress in handling missing data scenarios, current methodologies remain fundamentally constrained by their inability to effectively manage prolonged observation gaps and their low tolerance thresholds for high missing rates. In this work, we investigate the performance of reinforcement learning under the condition of missing data. We introduce a novel method, Mutual Information Aligned Generative Reinforcement Learning(MIA-GRL), which employs a spatiotemporal collaborative reconstruction to learn from historical reinforcement learning trajectories. This method synthesizes trajectories that encapsulate environmental characteristics and diversity through contextual information. We design an auxiliary loss function based on mutual information maximization, aiming to maximize the mutual information between the completed data and the original data, thereby ensuring that the completed data retains as much critical information from the original data as possible. Additionally, we utilize missing-aware contrastive learning to learn representations robust to missing patterns, enabling the policy network to capture intrinsic features relevant to the task. Experiments show our method outperforms state-of-the-art methods under the missing data conditions.

Conventional multi-agent reinforcement learning (MARL) methods that rely on flat cooperative structures often struggle to achieve intricate, hierarchical coordination, which is critical for effective multi-agent collaboration. To address this, we propose HACG, a framework for Hierarchical Graph Attention with Contribution-based Dynamic Grouping. Central to our approach is a dynamic grouping mechanism, which utilizes a sparsely regularized contribution generator to adaptively partition agents based on their learned contribution. A hierarchical graph attention network then processes these dynamic groups to foster both intra-group cooperation and inter-group coordination, enhancing global value function approximation. The entire framework is optimized end-to-end, jointly learning the grouping and cooperation policies. Experiments on the challenging SMAC and GRF benchmarks demonstrate that our method significantly outperforms state-of-the-art baselines.

In strategic decision-making tasks, determining how to assign limited costly resource towards the defender and the attacker is a central problem. However, it is hard for pre-allocated resource assignment to adapt to dynamic fighting scenarios, and exists situations where the scenario and rule of the Colonel Blotto (CB) game are too restrictive in real world. To address these issues, a support stage is added as supplementary for pre-allocated results, in which a novel two-stage competitive resource assignment problem is formulated based on CB game and stochastic Lanchester equation (SLE). Further, the force attrition in these two stages is formulated as a stochastic progress to consider the complex fighting progress, including the case that the player with fewer resources defeats the player with more resources and wins the battlefield. For solving this two-stage resource assignment problem, nested solving and noregret learning are proposed to search the optimal resource assignment strategies. Numerical experiments are taken to analyze the effectiveness of the proposed model and study the assignment strategies in various cases.

Multiple unmanned surface vehicles (USV) have demonstrated significant advantages through cooperation in complex maritime tasks. However, in open waters, multi-USV systems face dual risks arising from decision-making methodologies and environmental uncertainties, posing significant challenges to the robustness of cooperative policies. To address this challenge, this paper proposes a robust policy learning method ROMUC based on multi-agent reinforcement learning (MARL). Firstly, ROMUC addresses environmental uncertainties by learning the complete distribution of action values rather than scalar expected values. Secondly, it employs value average and operator switching methods to mitigate risks caused by inherent overestimation errors in decision-making approaches, thereby enhancing the robustness of cooperative policies for multi-USV. Finally, through testing in two typical mission scenarios (pursuit and denial) and comparisons with mainstream MARL methods, results demonstrate that the ROMUC approach effectively addresses dual challenges of environmental uncertainties and overestimation errors, significantly improving the robustness of cooperative policies.

Game theory plays a pivotal role in addressing Imperfect Information Games (IIGs), having given rise to the development of a suite of superhuman agents. However, constructing information sets and game trees in game theory approaches restricts the transferability of strategies. Recently, Reinforcement Learning (RL) has been introduced from the perfect information game to IIG, owing to its generalization capabilities. Yet, the challenges of efficient feature abstraction and stable learning within the RL paradigm persist. To address these challenges, we explore a stable actor-critic training framework with knowledge distillation based on supervised learning to enhance proximal policy optimization in strategy learning (MCA-PPO). For feature extraction, we engineer a multi-scale convolutional neural network that amalgamates local feature information across various receptive field sizes, unearthing the latent information embedded within potential feature sequences. We refine the network architecture further by integrating it with an attention mechanism, enabling it to discern relationships between feature channels and to autonomously identify the essence of multi-scale features. Regarding strategy learning, we employ the knowledge distillation technique to mitigate the instability inherent in proximal policy optimization, thereby hastening the progression of strategic learning. Finally, we conducted a systematic and comprehensive analysis of the MCA-PPO algorithm through a series of experiments on Teax Hold’em poker, showing the efficacy and transferability of the MCA-PPO framework.

Deep reinforcement learning (DRL) typically requires reinitializing training for new tasks, limiting its generalization due to isolated knowledge transfer. Meta-reinforcement learning (Meta-RL) addresses this by enabling rapid adaptation through prior task experiences, yet existing gradient-based methods like MAML suffer from poor out-of-distribution performance due to overfitting narrow task distributions. To overcome this limitation, we propose Evolving Gradient Regularization MAML (ER-MAML). By integrating evolving gradient regularization into the MAML framework, ER-MAML optimizes meta-gradients while constraining adaptation directions via a regularization policy. This dual mechanism prevents overparameterization and enhances robustness across diverse task distributions. Experiments demonstrate ER-MAML outperforms state-of-the-art baselines by 14.6% in out-of-distribution success rates. It also achieves strong online adaptation performance in the MetaWorld benchmark. These results validate ER-MAML's effectiveness in improving meta-RL generalization under distribution shifts.

The breakthrough and progress of intelligent gaming technology with deep reinforcement learning as the core in the field of games provide a method reference for the research of agents in sea-air wargames. The architecture design of the agent is the primary core key problem that needs to be solved, and a good architecture can reduce the complexity and difficulty of training and accelerate the convergence of policies. A stochastic game model of sea-air cross-domain cooperative decision-making has been proposed, and its corresponding equilibrium solution concepts have been analyzed. Based on the analysis of typical agent frameworks, aiming at the decision-making gaming process of sea-air wargames, and then an agent bi-level architecture based on multi-Agent hierarchical reinforcement learning is proposed, which can effectively solve the problems of collaboration and dimensional disaster. The key technologies are analyzed from four aspects: force coordination, agent network design, adversary modeling and training mechanism. Hoping to provide architectural guidance for the subsequent design and implementation of sea-air wargaming agents.

To solve the problem of long OODA cycle time and untimely decision-making by traditional decision-making means in air-sea confrontation and better comprehend advanced combat concepts worldwide, we propose a multi-agent reinforcement learning algorithm that can make decisions regarding our combat formations’ operations. The algorithm's state space, action space, and reward function are designed for a comprehensive battlefield situation, enabling it to solve the top-level decision-making problem for sea-air confrontation. The tactical execution of the decision-making results uses a military chess deduction platform to create a complete decision-making command pathway. Finally, we have conducted experiments to test the algorithm’s effectiveness and generalisation capability in an air-sea confrontation scenario.

Aiming at the intelligent design of combat systems,we propose a conceptual framework and application methods of Agent based on artificial intelligence technology.First,we analyze the concept of Agent and discuss the important signifi-cance of studying Agent in the combat system.Then,we introduce the research framework of artificial intelligence based on Agent,and multiple application methods of Agent in combat systems.Finally,we analyze the development trends of Agent technology and the risks and challenges that its combat applications may face.

Offline reinforcement learning (RL) has been widely used in safety-critical domains by avoiding dangerous and costly online interaction. A significant challenge is addressing uncertainties and risks outside of offline data. Risk-sensitive offline RL attempts to solve this issue by risk aversion. However, current model-based approaches only extract state transition information and reward information using dynamics models, which cannot capture risk information implicit in offline data and may result in the misuse of high-risk data. In this work, we propose a model-based offline policy optimization approach with a risk dynamics model (MOOR). Specifically, we construct a risk dynamics model using a quantile network that can learn the risk information of data, then we reshape model-generated data based on errors of the risk dynamics model and the risk information of data. Finally, we use a risk-averse algorithm to learn the policy on the combined dataset of offline and generated data. We theoretically prove that MOOR can identify risk information of data and avoid utilizing high-risk data, our experiments show that MOOR outperforms existing approaches and achieves state-of-the-art results in risk-sensitive D4RL and risky navigation tasks.

Reinforcement learning (RL) has become a dominant decision-making paradigm and has achieved notable success in many real-world applications, such as autonomous driving, robotic manipulation, robotic navigation and gaming AI. However, the powerful abilities of the RL variants were developed with advanced machine learning techniques. For example, deep neural networks play a crucial role in unlocking RL’s potential in large-scale decision-making tasks. Inspired by present day major successes of Transformer in natural language processing and computer vision, numerous bottlenecks have been overcome by combining Transformer with RL for decision-making. This paper presents a multiangle systematic survey of various Transformerbased RL (TransRL) models applied in decision-making tasks, including basic models, advanced algorithms, representative implementation instances, applications, and known challenges. Our work aims to provide insights into problems that inherently arise with the current RL approaches, and examines how we can address them with better TransRL models. To our knowledge, we are the first to present a comprehensive review of the recent Transformers research developments in RL for decisionmaking. We hope this survey provides a comprehensive review of TransRL models and also inspires the RL community in its pursuit of future directions. Finally, to keep track of the rapid TransRL developments in the decision-making domains, we summarize the latest relevant papers and their open-source implementations at https://github.com/williamyuanv0/Transformerin-Reinforcement-Learning-for-Decision-Making-A-Survey.

Meta-Reinforcement Learning (Meta-RL) is a machine learning paradigm aimed at learning reinforcement learning policies that can quickly adapt to unseen tasks with few-shot data. Nevertheless, applying Meta-RL to real-world applications faces challenges due to the cost of data acquisition. To address this problem, offline Meta-RL has emerged as a promising solution, focusing on learning policies from pre-collected data that can effectively and rapidly adapt to unseen tasks. In this paper, we propose a new offline Meta-RL method called Meta-Actor-Critic with Evolving Gradient Agreement (MACEGA). MACEGA utilizes an evolutionary approach to estimate meta-gradients conductive to generalization across unseen tasks. During meta-training, gradient evolution is utilized to meta-update the value network and policies. Moreover, we use gradient agreement as an optimization objective for meta-learning, thereby enhancing the generalization ability of the meta-policy. We experimentally demonstrate the robustness of MACEGA in handling offline data quality. Furthermore, extensive experiments on various benchmarks provide empirical evidence that MACEGA outperforms previous state-of-the-art methods in generalizing to unseen tasks, thus demonstrating its potential for real-world applications.