Multi-agent learning in mixed-motive coordination problems

Cooperation in settings where agents have different but overlapping preferences (mixed-motive settings) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied are simplistic in that they have a single cooperative outcome on which all agents can agree. Multi-agent systems in general may exhibit many payoff profiles which might be called cooperative, but which agents have different preferences over. This causes problems for independently trained agents that do not arise in the case of that there is a unique cooperative payoff profile. In this note, we illustrate this problem with a class of games called mixed-motive coordination problems (MCPs). We demonstrate the failure of several methods for achieving cooperation in sequential social dilemmas when used to independently train policies in a simple MCP. We discuss some possible directions for ameliorating MCPs.