We introduce DeepNash, an autonomous agent that plays the imperfect information game Stratego at a human expert level. Stratego is one of the few iconic board games that artificial intelligence (AI) has not yet mastered. It is a game characterized by a twin challenge: It requires long-term strategic thinking as in chess, but it also requires dealing with imperfect information as in poker. The technique underpinning DeepNash uses a game-theoretic, model-free deep reinforcement learning method, without search, that learns to master Stratego through self-play from scratch. DeepNash beat existing state-of-the-art AI methods in Stratego and achieved a year-to-date (2022) and all-time top-three ranking on the Gravon games platform, competing with human expert players.

The Game Theory & Multi-Agent team at DeepMind studies several aspects of multi-agent learning ranging from computing approximations to fundamental concepts in game theory to simulating social dilemmas in rich spatial environments and training 3-d humanoids in difficult team coordination tasks. A signature aim of our group is to use the resources and expertise made available to us at DeepMind in deep reinforcement learning to explore multi-agent systems in complex environments and use these benchmarks to advance our understanding. Here, we summarise the recent work of our team and present a taxonomy that we feel highlights many important open challenges in multi-agent research.

In this paper we investigate the Follow the Regularized Leader dynamics in sequential imperfect information games (IIG). We generalize existing results of Poincare recurrence from normal-form games to zero-sum two-player imperfect information games and other sequential game settings. We then investigate how adapting the reward (by adding a regularization term) of the game can give strong convergence guarantees in monotone games. We continue by showing how this reward adaptation technique can be leveraged to build algorithms that converge exactly to the Nash equilibrium. Finally, we show how these insights can be directly used to build state-of-the-art model-free algorithms for zero-sum two-player Imperfect Information Games (IIG).

The rapid progress in artificial intelligence (AI) and machine learning has opened unprecedented analytics possibilities in various team and individual sports, including baseball, basketball, and tennis. More recently, AI techniques have been applied to football, due to a huge increase in data collection by professional teams, increased computational power, and advances in machine learning, with the goal of better addressing new scientific challenges involved in the analysis of both individual players' and coordinated teams' behaviors. The research challenges associated with predictive and prescriptive football analytics require new developments and progress at the intersection of statistical learning, game theory, and computer vision. In this paper, we provide an overarching perspective highlighting how the combination of these fields, in particular, forms a unique microcosm for AI research, while offering mutual benefits for professional teams, spectators, and broadcasters in the years to come. We illustrate that this duality makes football analytics a game changer of tremendous value, in terms of not only changing the game of football itself, but also in terms of what this domain can mean for the field of AI. We review the state-of-the-art and exemplify the types of analysis enabled by combining the aforementioned fields, including illustrative examples of counterfactual analysis using predictive models, and the combination of game-theoretic analysis of penalty kicks with statistical learning of player attributes. We conclude by highlighting envisioned downstream impacts, including possibilities for extensions to other sports (real and virtual).

Multiplayer games have long been used as testbeds in artificial intelligence research, aptly referred to as the Drosophila of artificial intelligence. Traditionally, researchers have focused on using well-known games to build strong agents. This progress, however, can be better informed by characterizing games and their topological landscape. Tackling this latter question can facilitate understanding of agents and help determine what game an agent should target next as part of its training. Here, we show how network measures applied to response graphs of large-scale games enable the creation of a landscape of games, quantifying relationships between games of varying sizes and characteristics. We illustrate our findings in domains ranging from canonical games to complex empirical games capturing the performance of trained agents pitted against one another. Our results culminate in a demonstration leveraging this information to generate new and interesting games, including mixtures of empirical games synthesized from real world games.

We introduce and analyze a class of algorithms, called Mirror Ascent against an Improved Oppo-nent (MAIO), for computing Nash equilibria in two-player zero-sum games, both in normal form and in sequential form with imperfect information. These algorithms update the policy of each player with a mirror-ascent step to maximize the value of playing against an improved opponent. An improved opponent can be a best response, a greedy policy, a policy improved by policy gradient, or by any other reinforcement learning or search techniques. We establish a convergence result of the last iterate to the set of Nash equilibria and show that the speed of convergence depends on the amount of improvement offered by these improved policies. In addition, we show that under some condition, if we use a best response as improved policy, then an exponential convergence rate is achieved.

The role of automatic electrocardiogram (ECG) analysis in clinical practiceis limited by the accuracy of existing models. Deep Neural Networks (DNNs) aremodels composed of stacked transformations that learn tasks by examples. Thistechnology has recently achieved striking success in a variety of task andthere are great expectations on how it might improve clinical practice. Here wepresent a DNN model trained in a dataset with more than 2 million labeled examsanalyzed by the Telehealth Network of Minas Gerais and collected under thescope of the CODE (Clinical Outcomes in Digital Electrocardiology) study. TheDNN outperform cardiology resident medical doctors in recognizing 6 types ofabnormalities in 12-lead ECG recordings, with F1 scores above 80% andspecificity over 99%. These results indicate ECG analysis based on DNNs, previously studied in a single-lead setup, generalizes well to 12-lead exams, taking the technology closer to the standard clinical practice.

OpenSpiel is a collection of environments and algorithms for research in general reinforcement learning and search/planning in games. OpenSpiel supports n-player (single- and multi- agent) zero-sum, cooperative and general-sum, one-shot and sequential, strictly turn-taking and simultaneous-move, perfect and imperfect information games, as well as traditional multiagent environments such as (partially- and fully- observable) grid worlds and social dilemmas. OpenSpiel also includes tools to analyze learning dynamics and other common evaluation metrics. This document serves both as an overview of the code base and an introduction to the terminology, core concepts, and algorithms across the fields of reinforcement learning, computational game theory, and search.

L'invention concerne un systeme de stockage d'objets distribues (1) qui comprend plusieurs optimisations de performances concernant l'enregistrement de tres petits objets de donnees, de tres grands objets de donnees et de calculs CRC.

Designing an adaptive multi-agent system often requires the specification of interaction patterns between the different agents. To date, it remains unclear to what extent such interaction patterns influence the dynamics of the learning mechanisms inherent to each agent in the...

In this paper we introduce a mathematical model of naming games. Naming games have been widely used within research on the origins and evolution of language. Despite the many interesting empirical results these studies have produced, most of this research lacks a formal elucidating theory. In this paper we show how a population of agents can reach linguistic consensus, i.e. learn to use one common language to communicate with one another. Our approach differs from existing formal work in two important ways: one, we relax the too strong assumption that an agent samples infinitely often during each time interval. This assumption is usually made to guarantee convergence of an empirical learning process to a deterministic dynamical system. Two, we provide a proof that under these new realistic conditions, our model converges to a common language for the entire population of agents. Finally the model is experimentally validated.

Over the past years, several computational models have been introduced to study the coordination of communication between distributed agents. Although these models have given valuable insights into the mechanisms required for letting agents develop a successful communication system, few theoretical results have been obtained which substantiate these findings. In this paper we introduce a theoretical framework which allows us to analyze and compare different existing models in a uniform way. Therefore we only look at the observable behavior of an agent and not at the internal mechanisms that cause that behavior. In particular, we define an agent’s response function and argue that a stability analysis of its fixed points reveals crucial information about the convergence properties of the dynamical system of interacting agents.

There is a growing body of research on multi-agent systems bootstrapping a communication system. Most studies are based on simulation, but recently there has been an increased interest in the properties and formal analysis of these systems. Although very interesting and promising results have been obtained in these studies, they always rely on major simplifications. For example, although much larger populations are considered than was the case in most earlier work, previous work assumes the possibility of meaning transfer . With meaning transfer, two agents always exactly know what they are talking about. This is hardly ever the case in actual communication systems, as noise corrupts the agents’ perception and transfer of meaning. In this paper we first consider what happens when relaxing the meaning-transfer assumption, and propose a cross-situational learning scheme that allows a population of agents to still bootstrap a common lexicon under this condition. We empirically show the validity of the scheme and thereby improve on the results reported in (Smith, 2003) and (Vogt and Coumans, 2003) in which no satisfactory solution was found. It is not our aim to reduce the importance of previous work, instead we are excited by recent results and hope to stimulate further research by pointing towards some new challenges.

Recently there has been a growing interest in the properties and formal analysis of multi-agent systems bootstrapping a communication system (see e.g. .(Baronchelli et al., 2005) and the contribution of (De Vylder and Tuyls, 2005), this workshop.) Although very interesting and promising results were obtained in these studies, major simplifications were made. For example, although much larger populations are considered than was the case in most earlier work, both the cited works (1) assume the possibility of meaning transfer, i.e. the hearer always exactly knows the speaker’s intended meaning independent of whether he understands the speaker’s utterance and (2) only consider single-word utterances. In this draft we first consider what happens when relaxing the meaning-transfer assumption, and propose a cross-situational learning scheme that allows a population of agents to still bootstrap a common lexicon under this condition. We empirically show the validity of the scheme and thereby improve on the results reported in (Smith, 2003) and (Vogt and Coumans, 2003) in which no satisfactory solution was found. We then continue to identify some problems that arise when abandoning the single-word utterance simplification. However, no solid solutions are proposed for the identified problems. It is not our aim to reduce the importance of previous work, instead we are excited by recent results and hope to stimulate further research by pointing towards some new challenges. Draft prepared for the Workshop on Semiotic Dynamics of Language Games, Bagnovignoni (Siena Italy) 8-11

Does Language Shape the Way We Conceptualize the World? Joachim De Beule (joachim@arti.vub.ac.be) Vrije Universiteit Brussel, Artificial Intelligence Lab, Pleinlaan 2, 1050 Brussels, Belgium Bart De Vylder (bartdv@arti.vub.ac.be) Vrije Universiteit Brussel, Artificial Intelligence Lab, Pleinlaan 2, 1050 Brussels, Belgium mutually compatible conceptualization schemes or on- tologies. It is shown that, in turn, feedback on the com- municative success has to be propagated to the ontolog- ical level in order to obtain compatible ontologies. As such it is shown that a language both depends on and influences an agent’s ontology and vice versa. Abstract In this paper it is argued that the way the world is con- ceptualized for language is language dependent and the result of negotiation between language users. This is investigated in a computer experiment in which a popu- lation of artificial agents construct a shared language to talk about a world that can be conceptualized in multi- ple and possibly conflicting ways. It is argued that the establishment of a successful communication system re- quires that feedback about the communicative success is propagated to the ontological level, and thus that lan- guage shapes the way we conceptualize the world for communication. Related and Previous work There have been many computational models in which a population of artificial agents evolve a shared language [Cangelosi and Parisi, 2001]. Not so many however have discussed in depth the co-evolution of meaning and form. In the following two exceptions will be discussed briefly. Introduction and Research Question The Talking Heads Experiment Language and communication involve many aspects of human cognition including the sensory-motor schema’s needed to observe the world, the social abilities for es- tablishing joint attention and communicative intent and the mechanisms responsible for parsing and producing abstract grammatical expressions. A key issue here is how a population of distinct and only locally interacting agents (language users) can agree upon a global language. It is commonly accepted that at least part of the answer is self-organization: a con- sensus is reached through repeated peer-to-peer negoti- ations about how to express some meaning. A prerequisite for this, which is often neglected, is that the agents already have to agree upon the set of express- ible meanings. It is implicitly assumed that all agents conceptualize the world according to some universal on- tology. However, there are strong indications that the way in which observations are conceptualized for language is language dependent and also the result of negotiation be- tween language users. For example, different languages lexicalize color categories differently and it is suggested that color terms might have an influence on color cate- gorization (see for example [Steels and Belpaeme, 2005], [Roberson, 2005]; see also [Levinson, 2001] for evidence on how language appears to shape a language learner’s meaning structure.) We investigate this phenomenon in a population of artificial agents placed in an artificial world that can be conceptualized in multiple and conflicting ways. Agents are equipped with learning mechanisms that allow them to establish a shared language. A prerequisite for a suc- cessful communication system is that the agents have In the talking heads (TH) and related experiments (see e.g. [Steels, 1998]) a population of robots develop a shared ontology and lexicon to communicate about dif- ferently shaped and colored objects by playing language games. Each game two agents are presented with a col- lection of objects called the context. One of the objects is the topic of the game. Only one of the agents, the speaker, is informed about the topic. He conceptualizes the topic (i.e. construes a meaning describing the topic) and verbalizes the result. The other agent, the hearer, then should locate the topic. If he succeeds the game is a success , otherwise it is a failure. The current experiment is at a higher level of abstrac- tion and ignores many difficulties that arise when work- ing with real robots. This is done on purpose, as it allows us to precisely control the structure of the world and its influence on language. Also, the focus is here on the co-evolution of ontology and language. Although in the TH setup meaning and form co-evolve as well, there are some important differences. In the TH an ontological category is defined as a re- gion in some sensory channel. An example of a sensory channel is the horizontal position (HPOS) and an exam- ple of a ‘left’ category is 0≤HPOS<0.5. A new category is created by splitting a channel or an existing category in two. This is done when the agent fails to discriminate the topic from the other objects in the context. For ex- ample, the left category is not sufficient to discriminate the topic when the context contains two objects with a HPOS less than 0.5. This might be solved by subdivid- ing the left category into two subcategories etc. As such a so called discrimination tree is constructed.

In this paper it is argued that the way the world is conceptualized for language is language dependent and the result of negotiation between language users. This is investigated in a computer experiment in which a population of artificial agents construct a shared language to talk about a world that can be conceptualized in multiple and possibly conflicting ways. It is argued that the establishment of a successful communication system requires that feedback about the communicative success is propagated to the ontological level, and thus that language shapes the way we conceptualize the world for communication. Introduction and Research Question Language and communication involve many aspects of human cognition including the sensory-motor schema’s needed to observe the world, the social abilities for establishing joint attention and communicative intent and the mechanisms responsible for parsing and producing abstract grammatical expressions. A key issue here is how a population of distinct and only locally interacting agents (language users) can agree upon a global language. It is commonly accepted that at least part of the answer is self-organization: a consensus is reached through repeated peer-to-peer negotiations about how to express some meaning. A prerequisite for this, which is often neglected, is that the agents already have to agree upon the set of expressible meanings. It is implicitly assumed that all agents conceptualize the world according to some universal ontology. However, there are strong indications that the way in which observations are conceptualized for language is language dependent and also the result of negotiation between language users. For example, different languages lexicalize color categories differently and it is suggested that color terms might have an influence on color categorization (see for example [Steels and Belpaeme, 2005], [Roberson, 2005]; see also [Levinson, 2001] for evidence on how language appears to shape a language learner’s meaning structure.) We investigate this phenomenon in a population of artificial agents placed in an artificial world that can be conceptualized in multiple and conflicting ways. Agents are equipped with learning mechanisms that allow them to establish a shared language. A prerequisite for a successful communication system is that the agents have mutually compatible conceptualization schemes or ontologies. It is shown that, in turn, feedback on the communicative success has to be propagated to the ontological level in order to obtain compatible ontologies. As such it is shown that a language both depends on and influences an agent’s ontology and vice versa. Related and Previous work There have been many computational models in which a population of artificial agents evolve a shared language [Cangelosi and Parisi, 2001]. Not so many however have discussed in depth the co-evolution of meaning and form. In the following two exceptions will be discussed briefly. The Talking Heads Experiment In the talking heads (TH) and related experiments (see e.g. [Steels, 1998]) a population of robots develop a shared ontology and lexicon to communicate about differently shaped and colored objects by playing language games. Each game two agents are presented with a collection of objects called the context. One of the objects is the topic of the game. Only one of the agents, the speaker, is informed about the topic. He conceptualizes the topic (i.e. construes a meaning describing the topic) and verbalizes the result. The other agent, the hearer, then should locate the topic. If he succeeds the game is a success , otherwise it is a failure. The current experiment is at a higher level of abstraction and ignores many difficulties that arise when working with real robots. This is done on purpose, as it allows us to precisely control the structure of the world and its influence on language. Also, the focus is here on the co-evolution of ontology and language. Although in the TH setup meaning and form co-evolve as well, there are some important differences. In the TH an ontological category is defined as a region in some sensory channel. An example of a sensory channel is the horizontal position (HPOS) and an example of a ‘left’ category is 0≤HPOS<0.5. A new category is created by splitting a channel or an existing category in two. This is done when the agent fails to discriminate the topic from the other objects in the context. For example, the left category is not sufficient to discriminate the topic when the context contains two objects with a HPOS less than 0.5. This might be solved by subdividing the left category into two subcategories etc. As such a so called discrimination tree is constructed. It is clear that, in the absence of noise, an agent’s discrimination abilities can be made arbitrarily high by progressively growing the discrimination tree (i.e. introducing more specific categories.) However, this rapidly conflicts with the agent’s communicative success: the larger an agent’s ontology the larger the lexicon needed to express it and the less learnable the language will be. Therefore, when a certain discriminative success is reached no additional categories are created and infrequently used leaf-categories are deleted. An important point here is that all agent’s ontologies share a common structure. This is because all agents have the same sensor channels and they all use the same top-down mechanism to grow their ontology. This allows them to decide on which categories to prune and ensures, up to a certain level, that all remaining categories are relevant for all agents in the population (see also [Smith, 2003] for the influence of biased meaning creation on communication.) This is an important difference with the currently described experiment in which a category’s relevance in the population is not a priori known and it is not clear which categories to add or delete if multiple candidates are at hand. Color Category Formation In a more recent study ([Steels and Belpaeme, 2005]) it was already shown that feedback about the success in communicating color categories is most probably needed to explain the way in which humans categorize color for language. In this experiment the observation space is a continues real-valued 3 dimensional color-space and categories consist of prototypes in this space. Again, the more categories in the ontology the bigger its discriminative power will be but the less suitable for language. An important mechanism to control the size of an ontology here makes use of the fact that a distance measure can be defined between any two categories or observations. This allows for example to merge two categories that are close together by replacing them by their mean category. In the current experiment categories are more like predicates that are either true or false for an observed object or event (e.g. (red ?object) or (in-the-past ?event).) Merging such categories is not well defined and other means of controlling the size of the ontology are needed. In addition, in the color-category experiment categories can be shifted in color space. This is actually the main mechanism by which agents reach a consensus: if some agent defines the meaning of a word to be some point in color space, another agent can shift his own category associated with that word toward that point. Again, a shift operation is not at hand in the current experiment.

In this paper we introduce a mathematical model of naming games, named the sampling-amplification model. Naming games have been extensively used to investigate the dynamics of lexicon acquisition. Despite the many interesting empirical results these studies have produced, most of this research lacks a formal (realistic) elucidating theory. In this paper we try to bridge the gap between mathematical models and empirical studies of naming games in a novel manner, differing from existing work in two important ways: One, we relax the too strong assumption that the game is sampled infinitely often during each time interval. This assumption is usually made to guarantee convergence of an empirical learning process to a deterministic dynamical system. Although the dynamical system will predict the learning process approximately well, it cannot be considered a realistic setting as infinitely sampling does not occur in the real world. Two, we provide a proof that under these new realistic conditions, our model converges to a common language for the entire population of agents. Finally the model is experimentally validated.