This paper addresses the problem to find an optimal warning and intervention strategy for a partially autonomous driver’s assistance system. Here, an optimal strategy is regarded as the one minimizing the risk of collision with an obstacle ahead, while keeping the number of warnings and interventions as low as possible, in order to support the driver and avoid distraction or annoyance. A novel approach to this problem is proposed, based on the solution of a sequential decision making problem.

The FP7 EU project ISi-PADAS (Integrated Human Modelling and Simulation to support Human Error Risk Analysis of Partially Autonomous Driver Assistance Systems) endeavours to conceive an intelligent system called PADAS (Partially Autonomous Driver Assistance System) for aiding human drivers in driving safely by providing them with pertinent and accurate information in real time about the external situation and by acting as a co-pilot in emergency conditions. The system interacts with the driver through a Human–Machine Interface (HMI) installed on the vehicle using an adequate Warning and Intervention Strategy (WIS). In this paper, the design of the PADAS HMI as well as a decision-theoretic approach for deriving an optimal WIS are described.

Partially Autonomous Driver's Assistance Systems (PADAS) are systems aiming at providing a safer driving experience to people. Especially, one application of such systems is to assist the drivers in reacting optimally so as to prevent collisions with a leading vehicle. Several means can be used by a PADAS to reach this goal. For instance, warning signals can be sent to the driver or the PADAS can actually modify the speed of the car by braking automatically. An optimal combination of different warning signals together with assistive braking is expected to reduce the probability of collision. How to associate the right combination of PADAS actions to a given situation so as to achieve this aim remains an open problem. In this paper, the use of a statistical machine learning method, namely the reinforcement learning paradigm, is proposed to automatically derive an optimal PADAS action selection strategy from a database of driving experiments. Experimental results conducted on actual car simulators with human drivers show that this method achieves a significant reduction of the risk of collision.

The FP7 EU project ISi-PADAS aims at conceiving an intelligent system, called PADAS, to support drivers, which intervenes continuously from warning up to automatic braking in the whole longitudinal control of the vehicle. However, such supporting systems can have some unwanted side-effect: due to the presence of automation in the driving task, less attention and reaction are needed by the drivers to intervene in the longitudinal control of the vehicle. Such a paper aims at investigating the effects of the level of automation on drivers, in particular on their Situation Awareness, when the user is supported by a specific PADAS application, integrated with a driver's distraction classifier.

Decentralized planning in uncertain environments is a complex task generally dealt with by using a decision-theoretic approach, mainly through the framework of Decentralized Partially Observable Markov Decision Processes (DEC-POMDPs). Although DEC-POMDPS are a general and powerful modeling tool, solving them is a task with an overwhelming complexity that can be doubly exponential. In this paper, we study an alternate formulation of DEC-POMDPs relying on a sequence-form representation of policies. From this formulation, we show how to derive Mixed Integer Linear Programming (MILP) problems that, once solved, give exact optimal solutions to the DEC-POMDPs. We show that these MILPs can be derived either by using some combinatorial characteristics of the optimal solutions of the DEC-POMDPs or by using concepts borrowed from game theory. Through an experimental validation on classical test problems from the DEC-POMDP literature, we compare our approach to existing algorithms. Results show that mathematical programming outperforms dynamic programming but is less efficient than forward search, except for some particular problems.The main contributions of this work are the use of mathematical programming for DEC-POMDPs and a better understanding of DEC-POMDPs and of their solutions. Besides, we argue that our alternate representation of DEC-POMDPs could be helpful for designing novel algorithms looking for approximate solutions to DEC-POMDPs.

We consider the problem of tracking multiple, partially observed targets using multiple sensorsarranged in a given con guration. We model the problem as a special case of a ( nite horizon) DEC-POMDP. We present a quadratic program whose globally optimal solution yields an optimal trackingjoint policy, one that maximizes the expected targets detected over the given horizon. However, aglobally optimal solution to the QP cannot always be found since the QP is nonconvex. To remedythis, we present two linearizations of the QP to equivalent 0-1 mixed integer linear programs (MIPs)whose optimal solutions, which may be always found through the branch and bound method, forexample, yield optimal joint policies. Computational experience on di erent sensor con gurationsshows that nding an optimal joint policy by solving the proposed MIPs is much faster than usingexisting algorithms for the problem. 1 Introduction This paper addresses a special case of nite horizon DEC-POMDPs. The special case has been called anetwork distributed POMDP [4] or a factored DEC-POMDP [5]. Lately, this special case has receivedattention in these pages, especially for the problem of detecting multiple targets passing through a givencon guration of locations using multiple sensors, and specialized algorithms have been conceived for it[4], [6], [3]. Our focus too shall be on the multi-target tracking problem.In this problem, the set of agents (sensors) is partitioned into subsets. It is assumed that for eachsubset, we can de ne immediate rewards that are dependent on the actions of the agents of the subsetbut not on the actions of agents outside the subset. It is furthermore assumed that the probabilities withwhich an agent receives observations are independent of probabilities with which other agents receiveobservations. Finally, it is assumed that the probabilities of transitions between states are independentof actions of the agents.The purpose of the above partitioning scheme is to model autonomy for agents in one subset fromthose in other subsets. In the multi-target tracking problem (Figure 1), only the two sensors surroundingFigure 1: Sensor con gurations (reprised from [6]).1

This paper studies the problem of finding an optimal finite horizon joint policy for a decentralized partially observable Markov decision process (Dec-POMDP). We present a new algorithm for finding an optimal joint policy. The algorithm is based on the fact that the necessary condition for a joint policy to be optimal is that it be locally optimal (that is, a Nash equilibrium). Through the application of linear programming duality, the necessary condition can be transformed to a nonlinear program which can then further be transformed to a 0-1 mixed integer linear program (MILP) whose optimal solution is an optimal joint policy (in the sequence form). The proposed algorithm thus consists of solving this 0-1 MILP. Computational experience of the 0-1 MILP on two and three agent DEC-POMDPs gives mixed results. On some problems it is faster than existing algorithms, on others it is slower.

We consider the problem of finding an n-agent joint-policy for the optimal finite-horizon control of a decentralized Pomdp (Dec-Pomdp). This is a problem of very high complexity (NEXP-hard in n >= 2). In this paper, we propose a new mathematical programming approach for the problem. Our approach is based on two ideas: First, we represent each agent's policy in the sequence-form and not in the tree-form, thereby obtaining a very compact representation of the set of joint-policies. Second, using this compact representation, we solve this problem as an instance of combinatorial optimization for which we formulate a mixed integer linear program (MILP). The optimal solution of the MILP directly yields an optimal joint-policy for the Dec-Pomdp. Computational experience shows that formulating and solving the MILP requires significantly less time to solve benchmark Dec-Pomdp problems than existing algorithms. For example, the multi-agent tiger problem for horizon 4 is solved in 72 secs with the MILP whereas existing algorithms require several hours to solve it.

Nous nous interessons au probleme consistant a trouver une politique jointe optimale pour $n$ agents dans le cadre du controle optimal d'un processus decisionnel de Markov decentralise partiellement observe (Dec-POMDP). Le principe de notre approche est le suivant~: la politique jointe optimale d'un Dec-POMDP est equivalente a une politique \emph{sous-optimale} du POMDP lie, politique qui devrait en outre respecter des contraintes structurelles afin qu'elle puisse etre decentralisee. En s'appuyant sur ce principe, nous presentons un algorithme exact qui utilise la programmation lineaire mixte (PLM) pour trouver un vecteur de poids de realisation de sequences jointes (suite d'actions et d'observations jointes) qui represente ainsi une politique jointe. La politique jointe (decentralisable) optimale pour le Dec-POMDP derive directement de la solution de ce PLM. Des experimentation de notre algorithme sur des problemes de Dec-POMDP standards montrent qu'il est plus efficace (rapide) que les algorithmes exacts actuels de programmation dynamique.

We propose a formulation of a general-sum bimatrix game as a bipartite directed graph with the objective of establishing a correspondence between the set of the relevant structures of the graph (in particular elementary cycles) and the set of the Nash equilibria of the game. We show that finding the set of elementary cycles of the graph permits the computation of the set of equilibria. For games whose graphs have a sparse adjacency matrix, this serves as a good heuristic for computing the set of equilibria. The heuristic also allows the discarding of sections of the support space that do not yield any equilibrium, thus serving as a useful preprocessing step for algorithms that compute the equilibria through support enumeration.

This article presents the main reinforcement learning algorithms that aim at coordinating multi-agent systems by using tools and formalisms borrowed from Game Theory. Limits of these approaches are studied and discussed in order to draw some promising lines of research for that particular field. We argue more deeply around the central notions of Nash equilibrium and games with imperfect monitoring.

We consider the problem of learning strategy selection in games. The theoretical solution to this problem is a distribution over strategies that responds to a Nash equilibrium of the game. When the payoff function of the game is not known to the participants, such a distribution must be approximated directly through repeated play. Full knowledge of the payoff function, on the other hand, restricts agents to be strictly rational. In this classical approach, agents are bound to a Nash equilibrium, even when a globally better solution is obtainable. In this paper, we present an algorithm that allows agents to capitalize on their very lack of information about the payoff structure. The principle we propose is that agents resort to the manipulation of their own payoffs, during the course of learning, to find a ``game'' that gives them a higher payoff than when no manipulation occurs. In essence, the payoffs are considered an extension of the strategy set. At all times, agents remain rational vis-a-vis the information available. In self-play, the algorithm affords a globally efficient payoff (if it exists).

In this paper, we describe how certain aspects of the biological phenomena of stigmergy can be imported into multiagent reinforcement learning (MARL), with the purpose of better enabling coordination of agent actions and speeding up learning. In particular, we detail how these stigmergic aspects can be used to define an inter-agent communication framework.