The introduction of subtasks holds the promise of promoting coordination in scenarios without communication. Instead of manually defined subtasks, recent studies attempt to decompose the overall task and allocate subtasks to agents automatically, but it remains unclear how to acquire a set of proficient subtask representations. In essence, the subtasks serve as auxiliary signals that assist agents in deducing the broader context from limited observations. To embed maximal information into subtask representations, we propose to first learn a vector quantization variational autoencoder which takes individual observations of agents as inputs and reconstructs the global state based on their assigned subtasks as latent variables. Next, the informative representations can be readily integrated into various classic multi-agent reinforcement learning frameworks to facilitate insightful decisions of agents. Experiments on StarCraft II micro-war challenges and Google Research Football have demonstrated that our method learns reasonable and informative subtask representations, which facilitate the decision-making of agents and significantly improve the overall performance.

Value decomposition methods have gained popularity in the field of cooperative multi-agent reinforcement learning. However, almost all existing methods follow the principle of Individual Global Max (IGM) or its variants, which limits their problem-solving capabilities. To address this, we propose a dual self-awareness value decomposition framework, inspired by the notion of dual self-awareness in psychology, that entirely rejects the IGM premise. Each agent consists of an ego policy for action selection and an alter ego value function to solve the credit assignment problem. The value function factorization can ignore the IGM assumption by utilizing an explicit search procedure. On the basis of the above, we also suggest a novel anti-ego exploration mechanism to avoid the algorithm becoming stuck in a local optimum. As the first fully IGM-free value decomposition method, our proposed framework achieves desirable performance in various cooperative tasks.

The coordination between agents in multi-agent systems has become a popular topic in many fields. To catch the inner relationship between agents, the graph structure is combined with existing methods and improves the results. But in large-scale tasks with numerous agents, an overly complex graph would lead to a boost in computational cost and a decline in performance. Here we present DAGMIX, a novel graph-based value factorization method. Instead of a complete graph, DAGMIX generates a dynamic graph at each time step during training, on which it realizes a more interpretable and effective combining process through the attention mechanism. Experiments show that DAGMIX significantly outperforms previous SOTA methods in large-scale scenarios, as well as achieving promising results on other tasks.

Almost all multi-agent reinforcement learning algorithms without communication follow the principle of centralized training with decentralized execution. During the centralized training, agents can be guided by the same signals, such as the global state. However, agents lack the shared signal and choose actions given local observations during execution. Inspired by viewpoint invariance and contrastive learning, we propose consensus learning for cooperative multi-agent reinforcement learning in this study. Although based on local observations, different agents can infer the same consensus in discrete spaces without communication. We feed the inferred one-hot consensus to the network of agents as an explicit input in a decentralized way, thereby fostering their cooperative spirit. With minor model modifications, our suggested framework can be extended to a variety of multi-agent reinforcement learning algorithms. Moreover, we carry out these variants on some fully cooperative tasks and get convincing results.

Role-based multi-agent reinforcement learning (MARL) holds the promise of achieving scalable multi-agent cooperation by decomposing complex tasks through the concept of roles and has enjoyed great success in various tasks. However, conventional role-based MARL methods typically assign a single role to each agent, limiting the agent's behavior in certain scenarios. In real life, an individual usually performs multiple responsibilities in a given task. To meet such situations, we propose a novel soft role assignment (SORA) process that enables an agent to play multiple roles simultaneously. Concretely, SORA first generates a role distribution via the attention mechanism to interpret the agent's identity as a combination of different roles. To ensure consistent behavior with an agent's assigned role, we also introduce role-specific Q networks for decision-making. By virtue of these advances, our proposed method makes a prominent improvement over the prior state-of-the-art approaches on StarCraft multi-agent challenges and Google Research Football.