What is the Solution for State-Adversarial Multi-Agent Reinforcement Learning?
arxiv(2022)
摘要
Various methods for Multi-Agent Reinforcement Learning (MARL) have been
developed with the assumption that agents' policies are based on accurate state
information. However, policies learned through Deep Reinforcement Learning
(DRL) are susceptible to adversarial state perturbation attacks. In this work,
we propose a State-Adversarial Markov Game (SAMG) and make the first attempt to
investigate different solution concepts of MARL under state uncertainties. Our
analysis shows that the commonly used solution concepts of optimal agent policy
and robust Nash equilibrium do not always exist in SAMGs. To circumvent this
difficulty, we consider a new solution concept called robust agent policy,
where agents aim to maximize the worst-case expected state value. We prove the
existence of robust agent policy for finite state and finite action SAMGs.
Additionally, we propose a Robust Multi-Agent Adversarial Actor-Critic (RMA3C)
algorithm to learn robust policies for MARL agents under state uncertainties.
Our experiments demonstrate that our algorithm outperforms existing methods
when faced with state perturbations and greatly improves the robustness of MARL
policies. Our code is public on
https://songyanghan.github.io/what_is_solution/.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要