Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning

Linear value decomposition is a widely-used structure to scale up multi-agent Q-learning algorithms in cooperative settings. To develop a deeper understanding of this popular technique, this paper provides the first theoretical analysis to characterize its internal mechanism. Our main results reveal two novel insights: (1) Linear value decomposition structure implicitly implements a classical credit assignment called difference rewards. (2) This implicit credit assignment requires on-policy data distribution to achieve numerical stability. In the empirical study, our experiments also demonstrate that most deep multi-agent Q-learning algorithms using linear value decomposition structure cannot efficiently utilize off-policy samples.