Multi-agent reinforcement learning (MARL) has achieved notable progress in cooperative tasks, yet the common use of parameter sharing often leads to behavioral homogeneity, which restricts the diversity and adaptability of agent coordination. To address this issue, we propose Contrastive Learning for Role Representation (CLRE), a framework designed to promote strategic diversity by encouraging agents to learn distinct and semantically meaningful roles. The key idea is to employ a Large Language Model (LLM) as an offline domain expert that automatically generates semantic role labels from agent states. These labels provide reliable supervision for a contrastive learning objective, guiding the model to acquire role embeddings that are both discriminative and interpretable. To ensure these role representations effectively support coordination, CLRE incorporates a graph attention network that captures inter-agent relationships and a dual-attention value function that integrates role information with the global state for context-aware decision-making. Experiments on the StarCraft Multi-Agent Challenge (SMAC) benchmark demonstrate that CLRE achieves superior win rates and sample efficiency compared with state-of-the-art MARL methods. Ablation studies further confirm that each component contributes to the overall performance, showing that the proposed approach enables diverse and efficient cooperation among agents in complex environments.
This paper introduces a novel approach to advancing distributed cooperative learning in multi-agent systems (MASs) using Gaussian Process (GP) regression. The primary contribution lies in the development of a prior-aware selective distributed GPs (Pri-GP) algorithm, which allows agents to selectively collaborate by requesting predictions from neighboring agents based on their trustworthiness. This selective learning mechanism ensures that agents can avoid relying on neighbors with potentially flawed prior knowledge, thereby improving prediction accuracy, particularly in scenarios where an agent’s prior information is incorrect or outdated. Moreover, Pri-GP eliminates the need for costly variance calculations typically required for determining aggregation weights. We also extend this framework to the online-Pri-GP algorithm, which integrates real-time learning strategies, demonstrating superior performance in dynamical systems by achieving lower prediction errors. Additionally, we provide a prediction error bound within the Pri-GP framework, offering a guarantee of the aggregated predictions. Simulation results demonstrate that Pri-GP and online-Pri-GP outperform existing distributed GPs approaches in function approximation and system identification.
For cover function games (CoFGs), we address the problem of organising agents into possibly overlapping coalitions so as to maximise social welfare. We introduce a subclass of CoFGs called probabilistically monotone CoFGs. Monotonicity means that the closer a cover is to an optimum, the higher its welfare. Probabilistic monotonicity means that monotonicity is satisfied with some probability, i.e., some violations of monotonicity are permitted as long as the number of violations is bounded. In addition, externalities are permitted and social welfare is not restricted to the sum function. For such games, we obtain a bound on the number of monotonicity violations that can be permitted for computing an exact optimum. For probabilistically monotone CoFGs with a bound on the number of monotonicity violations, we devise algorithms for computing an exact optimum and analyze their time complexities. We also provide constructive proofs which form the basis for our algorithms. Placing our algorithms in the context of the existing literature, we note that a key unique feature of our algorithms is that they do not require the numeric welfare values of covers as input, rather, only an ordering over these values is required. Moreover, the ordering that is required as input does not have to be the actual ordering over the welfare values, it can be an approximate anticipated version of the actual ordering. The anticipated ordering is allowed to differ from the actual ordering in that anticipation errors, captured by monotonicity violations, are permitted in the anticipated ordering as long as the number of errors is bounded. Our algorithms are therefore highly relevant for practical applications like multi-agent task allocation where welfare values are only revealed after coalition formation although an ordering over the values can be approximately anticipated in advance of coalition formation.
Indirect reciprocity, supported by simple reputation assessment and social norms, has been demonstrated as an effective mechanism for enabling cooperation in populations of self-interested individuals. However, it has been shown that where there is noise in the performance of actions, or in observers’ perceptions, cooperation may not emerge. Higher-order social norms and generosity have been investigated as potential mechanisms to support cooperation in such environments, but are ineffective without additional, limiting, assumptions. In particular, higher-order norms have typically been investigated in cases where reputation is binary (‘good’ or ‘bad’) and where all agents ascribe the same reputation to an individual, implying full and perfect observation of actions. Generosity with an ‘aligned’ strategy, where all agents have the same likelihood of being generous, has been shown to be ineffective where reputation is binary. In this work we consider reciprocity emergence mechanisms in noisy domains and where agent observations may be incomplete or inaccurate. Our hypothesis is that nuanced reputation scores will enable generosity to be more effective, since an individual act of generosity will have a less extreme impact on reputation. We also investigate whether replacing the ‘aligned’ setting for generosity with a ‘non-aligned’ alternative, which we refer to as forgiveness, will support cooperation in noisy partially observable environments, without the level of ‘unjustified benevolence’ exhibited by generosity. We show both analytically and empirically that generosity when combined with fine grained reputation can help cooperation emerge, and that forgiveness can support cooperation in certain settings.
Ensuring the safety of reinforcement learning (RL) policies in high-stakes environments requires more than formal verification: it needs interpretability and targeted falsification—the deliberate search for counter-examples that expose potential failures before deployment. We present AEGIS-RL (Abstract, Explainable Graphs for Integrated Safety in RL), a hybrid framework that unifies (1) explainable RL, (2) probabilistic model checking, and (3) risk-guided falsification, and augments them with (4) a lightweight runtime safety shield that switches to a fallback policy when estimated risk exceeds a threshold. AEGIS-RL first builds a directed, semantically meaningful graph from offline trajectories that blends local and global explanations to make policy behavior transparent and verifier-friendly. This abstract graph is fed to a probabilistic model checker (e.g., Storm) to verify temporal safety specifications; when violations exist, the checker returns interpretable counterexample traces that pinpoint how the policy fails. When specifications appear satisfied, AEGIS-RL estimates residual risk during checking to steer falsification toward high-risk, under-explored states, broadening coverage beyond the offline data. Across safety-critical benchmarks including two MuJoCo tasks and a medical insulin-dosing scenario; AEGIS-RL uncovers significantly more violations than uncertainty- and fuzzing-based baselines and yields a broader, more novel set of failure trajectories. The resulting explanations and counterexamples provide actionable guidance to understand, debug, and repair unsafe policies while enabling runtime mitigation without retraining.
We revisit the computational problem of partitioning indivisibles into bundles among alternatives to maximize value (e.g., welfare). These problems have broad applications, yet many important variants are computationally hard, including well-known instances in operations research, computational economics, and artificial intelligence. To address this complexity, we analyze novel restrictions and concise representations for this problem class and establish new complexity results. Building on these findings, we present improved complexity bounds using a hypergraph-based characterization and introduce a novel “bootstrapped” dynamic programming method that significantly outperforms existing algorithms for a broad class of problems. Other findings include: polynomial-time solvability for problems with non-negative synergies and two alternatives; the problem remaining NP-hard even when bounding bundle sizes to two, with other instances being polynomial-time solvable; and exploration of bounds for more general cases allowing externalities and balanced (mixed) welfare, offering efficient approximation and non-trivial exponential-time algorithms for many hard cases.
Vehicle Routing Problems (VRPs) involve multi-agent route optimization, with the objective of targeting optimal routes for a fleet of vehicles to serve a set of customers. Existing neural solvers based on the divide-and-conquer approach for VRPs in general, and capacitated VRP (CVRP) in particular, integrate the global partition of an instance with the local construction for each resulting subinstance to enhance generalization. However, during the global partition phase, misclusterings within subgraphs have a tendency to progressively compound throughout the multi-step decoding process of the learning-based partition policy. This suboptimal behavior of the partition policy, in turn, may lead to a dramatic deterioration in the performance of the overall decomposition-based system, despite using optimal local constructions. To address these challenges, we propose a versatile Hierarchical Learning-based Graph Partition (HLGP) framework, which is tailored to benefit the partition of CVRP instances by synergistically integrating global and local partition policies. Specifically, the global partition policy is tasked with creating a coarse multi-way partition to generate a sequence of simpler two-way partition subtasks. These subtasks mark the initiation of the subsequent K local partition levels. At each local partition level, subtasks exclusive to this level are assigned to the local partition policy which benefits from the insensitive local topological features to incrementally alleviate the compounded errors. This framework is versatile in the sense that it optimizes the involved partition policies towards a unified objective, which is harmoniously compatible with both reinforcement learning (RL) and supervised learning (SL) paradigms. Additionally, we decouple the synchronized training into individual training of each component to circumvent the instability issue. Furthermore, we point out the importance of treating the subproblems encountered during the partition process as individual training instances. Extensive experiments conducted on various CVRP benchmarks demonstrate the effectiveness and generalization capabilities of the HLGP framework under both scale and distribution shifts. The source code is available at https://github.com/panyxy/hlgp_cvrp .
We study the mechanism design problem of facility location on a metric space in the learning-augmented framework, where mechanisms have access to imperfect predictions of the optimal facility locations. Our objective is to design strategyproof (SP) mechanisms that truthfully elicit agents’ preferences over facility locations and, using the given prediction, select a facility location that approximately minimizes the maximum cost among all agents. In particular, we seek SP mechanisms whose approximation guarantees depend on the prediction error: they should achieve improved performance when the prediction is accurate (the property of consistency) while still ensuring strong worst-case guarantees when the prediction is arbitrarily inaccurate (the property of robustness). On the real line, we characterize all deterministic SP mechanisms with consistency strictly better than 2 and bounded robustness for the maximum cost. We show that any such mechanism must coincide with the MinMaxP mechanism, which returns the prediction if it lies between the two extreme agent locations and otherwise returns the agent location closest to the prediction. For any prediction error η≥ 0 , we prove that MinMaxP achieves a (1+min (1, η )) -approximation and that no deterministic SP mechanism can obtain a better approximation ratio. In addition, for two-dimensional spaces with the ℓ ^p distance, we analyze the approximation guarantees of a deterministic mechanism that applies MinMaxP independently on each coordinate, as well as a randomized mechanism that selects between two deterministic mechanisms with carefully chosen probabilities. We further extend these results to the L_p -norm social cost objective on the line metric and the maximum cost objective on the tree metric. Finally, we examine the group strategyproofness of the mechanisms.
We present ε -retrain, a general exploration strategy for reinforcement learning (RL) that encourages adherence to behavioral preferences while preserving the convergence guarantees of the underlying RL algorithm. ε -retrain maintains a dynamic collection of retrain areas—regions of the state space where the agent previously violated a specified preference—and mixes the standard uniform restart distribution with states from these areas, according to a decaying parameter ε . This mixed retraining thus focuses on enforcing the desired behaviors in the collected areas. We develop the theory for both policy and value-based methods, showing that: (i) in policy-based settings, our method retains monotonic improvement bounds; and (ii) in value-based settings, ε -retrain preserves convergence properties without additional assumptions. The approach is simple to integrate into existing RL algorithms and improves sample efficiency and behavioral adherence in the locomotion, power systems, and navigation tasks tested. These results establish ε -retrain as a lightweight, theoretically grounded mechanism for incorporating behavioral preferences into RL.
A wide range of real-world applications can be formulated as Multi-Agent Path Finding (MAPF) problem, where the goal is to find collision-free paths for multiple agents with individual start and goal locations. State-of-the-art MAPF solvers are mainly centralized and rely on global information, which limits their scalability and flexibility when facing changes or new maps that require expensive replanning. Multi-agent reinforcement learning (MARL) offers an alternative approach to addressing MAPF problems by learning decentralized policies that generalize across a variety of maps. While there exist some prior works that attempt to connect both areas, the proposed techniques are heavily engineered and very complex due to the integration of many mechanisms that limit generality and are expensive to use. We argue that much simpler and more general approaches are needed to enable decentralized MAPF in a sustainable manner at significantly lower cost. In this paper, we propose Confidence-based Auto-Curriculum for Team Update Stability (CACTUS) as a lightweight MARL approach to decentralized MAPF. CACTUS defines a simple reverse curriculum scheme, where the goal of each agent is randomly placed within an allocation radius around the agent’s start location. The allocation radius increases gradually as all agents improve, which is assessed by a confidence-based measure. In addition, we propose an extension called Confidence- and Conflict-Based Curriculum Learning with Allocation Radius Adaptation (C ^3 LARA), using weighted sampling of goal locations to improve conflict resolution in scenarios of high agent density. We provide a theoretical analysis of the strengths and limitations of CACTUS regarding exploration efficiency, optimality, and multi-agent coordination. We evaluate CACTUS and C ^3 LARA across various maps of different sizes, obstacle densities, and numbers of agents. Our experiments demonstrate better performance and generalization capabilities than state-of-the-art MARL approaches with less than 600,000 trainable parameters, which is less than 5
Automated Negotiation (AN) is an approach for reaching agreements between self-interested agents. Early work in automated negotiation focused on proving game-theoretic results for simplified negotiation scenarios (e.g. when all agents’ preferences are common knowledge). More recent research - with some exceptions - focuses on developing novel mediated negotiation mechanisms or effective strategies for the Alternating Offers Protocol (AOP) and its variations. Recently, we proposed the Tentative Acceptance Unique offers (TAU) protocol as an alternative to AOP with promising empirical results (Mohammad 2023). In this paper, we develop a Perfect Bayesian Equilibrium (PBE) strategy for TAU in bilateral negotiations with no information about partner preferences. We analyze its theoretical properties showing that it renders TAU exactly complete, and Pareto-optimal for all bilateral discrete negotiation scenarios, and fair in the sense of a modified Kalai criterion for almost all bilateral discrete negotiation scenarios. Moreover, we empirically compare TAU combined with the proposed strategy against AOP with its state-of-the-art strategies and show empirically that it provides a higher expected advantage for all agents while achieving higher agreement rates, Pareto-optimality, fairness, and welfare on widely used negotiation scenarios.
Traditional Agent-Oriented Software Engineering (AOSE) methods rely on explicit and expert-driven design for Multi-Agent Systems (MAS), but often lack automation. In contrast, Multi-Agent RL (MARL) and related fields offer automated ways to model environments and learn suitable agent policies. However, integrating these techniques into AOSE remains underexplored partly due to the lack of control, explainability, and unifying frameworks. We propose MOISE+MARL Assisted MAS Design (MAMAD), a four-activity method framing MAS design as a constrained optimization problem: learning joint policies that maximize rewards while respecting ℳOISE^+ roles and goals. The activities include: 1) Modeling the environment, 2) Training under organizational constraints, 3) Analyzing emergent behaviors, 4) Transferring to real-world deployment. We evaluate MAMAD on various environments, showing that the generated MAS exhibit expected performance, compliance with design requirements and are explainable, while reducing manual design overhead.
We present an approach for the quantification of the usefulness of transfer in reinforcement learning via regret bounds for a multi-agent setting. Considering a number of ℵ agents operating in the same Markov decision process, however possibly with different reward functions, we consider the regret each agent suffers with respect to an optimal policy maximizing its average reward. We show that when the agents share their observations the mutual regret of all agents is smaller by a factor of √(ℵ) compared to the case when each agent has to rely on the information collected by itself. This result demonstrates how considering the regret in multi-agent settings can provide theoretical bounds on the benefit of sharing observations in transfer learning.
Task delegation is a fundamental mechanism adopted by agents to solve problems that involve teamwork. A critical issue in this context is trust establishment, in which agents must estimate the trustworthiness of potential partners based on their social behavior and environmental conditions. In the literature, most computational trust models address task delegation from a mono-episodic perspective, ignoring the possibility of sub-delegations and the resulting formation of delegation chains. Delegation chains enable the representation of complex social structures that capture agents’ dependency relationships. This work presents a task delegation model that explicitly supports sub-delegations through task decomposition and recursive delegation, while accounting for delegation chains. In the proposed model, agents select partners based on historical information about their performance, combined with social evaluations such as social image, reputation, and references. Experimental results show that, when compared to mono-episodic delegation approaches, the proposed model is particularly effective in dynamic environments where agents’ behavior may change over time.
Multi-objective reinforcement learning (MORL) algorithms tackle sequential decision problems where agents may have different preferences over (possibly conflicting) reward functions. These algorithms often learn a set of policies, each optimized for a particular agent preference, that are later reused when optimizing policies for different preferences. We introduce a novel algorithm that builds upon Generalized Policy Improvement (GPI) to construct principled, formally-derived prioritization schemes that improve sample efficiency. These correspond to active-learning strategies by which the agent can identify (i) the most promising preferences/objectives to train on at each moment; and (ii) the most relevant previous experiences to learn policies for new agent preferences through a novel Dyna-style MORL method. We prove our algorithm is guaranteed to always converge to an optimal solution in a finite number of steps, or an ϵ -optimal solution (for a bounded ϵ ) if the agent can only identify sub-optimal policies. Our method monotonically improves the quality of its partial solutions while learning. We also introduce a bound that characterizes the maximum utility loss (with respect to the optimal solution) incurred by intermediate policies identified by our method during learning. Finally, we propose a novel epistemic uncertainty-aware extension of GPI that exploits high-confidence lower bounds to mitigate the impact of unreliable action-value estimates in GPI policies, and prove that it provides tighter performance bounds than the current state of the art. We empirically show that our method outperforms state-of-the-art MORL algorithms in challenging multi-objective tasks.
Opponent models that predict opponents’ utility functions can help achieve favorable outcomes in automated bilateral multi-issue negotiations. Bayesian learning-based opponent models are flexible and adaptable to various negotiation contexts. However, existing Bayesian learning-based opponent models compromise prediction accuracy for computational efficiency by assuming independent issues and specific utility function shapes. We propose a novel Bayesian learning-based opponent model that improves prediction accuracy while maintaining computational efficiency by relaxing the shape assumption and separately learning each parameter of the utility function. Although parameters are estimated independently, this removes structural constraints and increases mutual dependence between parameters during inference. Each parameter is estimated with its conditional expectation, conditioned on the estimates of the other parameters, and computed efficiently through an iterative learning algorithm. We further introduce a resampling method to mitigate degeneracy in the hypothesis space and maintain diversity. Experiments across 45 negotiation domains against seven temporal and 10 Automated Negotiating Agents Competition (ANAC) final-list agents show that the proposed model outperforms existing Bayesian learning-, frequency-, and value-based opponent models. Ablation results validate the effectiveness and synergy of the parametric interrelation consideration and the resampling method.
Hybrid Intelligence (HI) is an emerging paradigm in which artificial intelligence (AI) augments human intelligence. The current literature lacks systematic models that guide the design and evaluation of HI systems. Further, discussions around HI primarily focus on technology, neglecting the holistic human-AI ensemble. In this paper, we take the initial steps toward the development of a quality model for characterizing and evaluating HI systems from a human-AI teams perspective. We first conducted a study investigating the adequacy of properties commonly associated with effective human teams to describe HI. The study features the insights of 50 HI researchers, and shows that various human team properties, including boundedness, interdependence, competency, purposefulness, initiative, normativity, and effectiveness, are important for HI systems. Based on these results, we developed a quality model for HI teams composed of seven high-level quality attributes, further refined into 16 specific ones. To evaluate the relevance and understanding of the proposed attributes, we conducted a second empirical investigation by staging competitions in which participants used the quality model to develop and analyze HI usage scenarios. Our analysis of 48 collected scenarios, which we openly release, confirms the proposed attributes’ relevance and highlights insights that emerge when designers consider the quality model in HI system design.
Multi-agent systems are designed to deal with open, distributed systems with unpredictable dynamics, which makes them inherently hard to test. The value of using simulation for this purpose is recognized in the literature, although achieving sufficient fidelity (i.e., the degree of similarity between the simulation and the real-world system) remains a challenging task. This is exacerbated when dealing with cognitive agent models, such as the Belief Desire Intention (BDI) model, where the agent codebase is not suitable to run unchanged in simulation environments, thus increasing the reality gap between the deployed and simulated systems. We argue that BDI developers should be able to test in simulation the same specification that will be later deployed, with no surrogate representations. Thus, in this paper, we discuss how the control flow of BDI agents can be mapped onto a Discrete Event Simulation (DES), showing that such integration is possible at different degrees of granularity. We substantiate our claims by producing an open-source prototype integration between two pre-existing tools (JaKtA and Alchemist), showing that it is possible to produce a simulation-based testing environment for distributed BDI} agents, and that different granularities in mapping BDI agents over DESs may lead to different degrees of fidelity.
