Robots working collaboratively can share observations with others to improve team performance, but communication bandwidth is limited. Recognizing this, an agent must decide which observations to communicate to best serve the team. Accurately estimating the value of a single communication is expensive; finding an optimal combination of observations to put in the message is intractable. In this paper, we present OCBC, an algorithm for Optimizing Communication under Bandwidth Constraints. OCBC uses forward simulation to evaluate communications and applies a bandit-based combinatorial optimization algorithm to select what to include in a message. We evaluate OCBC’s performance in a simulated multi-robot navigation task. We show that OCBC achieves better task performance than a state-of-the-art method while communicating up to an order of magnitude less.

This chapter presents multi-policy decision-making (MPDM): a novel approach to navigating in dynamic multi-agent environments. Rather than planning the trajectory of the robot explicitly, the planning process selects one of a set of closed-loop behaviors whose utility can be predicted through forward simulation that captures the complex interactions between the actions of these agents. These polices capture different high-level behavior and intentions, such as driving along a lane, turning at an intersection, or following pedestrians. We present two different scenarios where MPDM has been applied successfully: an autonomous driving environment models vehicle behavior for both our vehicle and nearby vehicles and a social environment, where multiple agents or pedestrians configure a dynamic environment for autonomous robot navigation. We present extensive validation for MPDM on both scenarios, using simulated and real-world experiments.

Risk-aware Multi-Policy Decision Making (MPDM) is a powerful framework for reliable navigation in a dynamic social environment where rather than evaluating individual trajectories, a "library" of policies (reactive controllers) is evaluated by anticipating potentially dangerous future outcomes using an on-line forward roll-out process. There is a core tension in Multi-Policy Decision Making (MPDM) systems - it is desirable to add more policies to the system for flexibility in finding good policies, however, this increases computational cost. As a result, MPDM was limited to small (perhaps 5-10) discrete policies - a significant performance bottleneck. In this paper, we radically enhance the expressivity of MPDM by allowing policies to have continuous-valued parameters, while simultaneously satisfying real-time constraints by quickly discovering promising policy parameters through a novel iterative gradient-based algorithm. Our evaluation includes results from extensive simulation and real-world experiments in semi-crowded environments.

In Multi-Policy Decision-Making (MPDM), many computationally-expensive forward simulations are performed in order to predict the performance of a set of candidate policies. In risk-aware formulations of MPDM, only the worst outcomes affect the decision making process, and efficiently finding these influential outcomes becomes the core challenge. Recently, stochastic gradient optimization algorithms, using a heuristic function, were shown to be significantly superior to random sampling. In this paper, we show that accurate gradients can be computed - even through a complex forward simulation - using approaches similar to those in deep networks. We show that our proposed approach finds influential outcomes more reliably, and is faster than earlier methods, allowing us to evaluate more policies while simultaneously eliminating the need to design an easily-differentiable heuristic function. We demonstrate significant performance improvements in simulation as well as on a real robot platform navigating a highly dynamic environment.

In the Multi-Policy Decision Making (MPDM) framework, a robot's policy is elected by sampling from the distribution of current states, predicting future outcomes through forward simulation, and selecting the policy with the best expected performance. Electing the best plan depends on sampling initial conditions with influential (very high costs) outcomes. Discovering these configurations through random sampling may require drawing many samples, which becomes a performance bottleneck. In this paper, we describe a risk-aware approach which augments this sampling with an optimization process that helps discover those influential outcomes. We describe how we overcome several practical difficulties with this approach, and demonstrate significant performance improvements on a real robot platform navigating a semi-crowded, highly dynamic environment.

In dynamic environments crowded with people, robot motion planning becomes difficult due to the complex and tightly-coupled interactions between agents. Trajectory planning methods, supported by models of typical human behavior and personal space, often produce reasonable behavior. However, they do not account for the future closed-loop interactions of other agents with the trajectory being constructed. As a consequence, the trajectories are unable to anticipate cooperative interactions (such as a human yielding), or adverse interactions (such as the robot blocking the way). In this paper, we propose a new method for navigation amongst pedestrians in which the trajectory of the robot is not explicitly planned, but instead, a planning process selects one of a set of closed-loop behaviors whose utility can be predicted through forward simulation. In particular, we extend Multi-Policy Decision Making (MPDM) [1] to this domain using the closed-loop behaviors Go-Solo, Follow-other, and Stop. By dynamically switching between these policies, we show that we can improve the performance of the robot as measured by utility functions that reward task completion and penalize inconvenience to other agents. Our evaluation includes extensive results in simulation and real-world experiments.

A long standing goal in artificial intelligence has been to develop rational agents that can reliably interact with the environment and perform interesting tasks. While traditional control theory enables us to efficiently develop agents for environments whose dynamics can be modelled with reasonable accuracy, in many real world problems the dynamics of the environment is unknown to the agent due to the inherent complexity of the environment. A common approach is to model this problem as a Markov Decision Process (MDP). In reinforcement learning (RL) algorithms, the agent explores the environment to gain knowledge about it while simultaneously trying to maximize the accumulated reward it receives. Large scale RL problems with continuous state-action spaces pose several challenges. They require generalizing the experience obtained by the agent by tedious feature selection for function approximation. For real-time applications, the RL algorithm must also ensure efficient action selection. Lastly, they must be computationally efficient. This project investigates and extends a recent algorithm designed to address this issue. Non-parametric approximate linear programming (NP-ALP) is a sample-based approach to value function approximation for MDPs with continuous stateand action-spaces, which requires only a distance function and an estimate of the Lipschitz constant of the value function. Using these inputs and a set of training data, NP-ALP finds a value function that is consistent with the specified Lipschitz constant. Previous theoretical work provided appealing sample

Supply-chain management problems are quite common in various industries and it is becoming increasingly necessary to tackle uncertainties while making decisions due to the rapid rise in production and consumption levels, and shortening of product life cycles. In our work, we tackle this problem of general stochastic supply-chain management problem by formulating it as a multi-arm non-contextual bandit problem and then taking a policy gradient descent approach (a Reinforcement Learning approach) to find a robust policy. The gradient descent is guided by cost from a simulator which models the demand, lead times and other uncertainties. Our experiments demonstrate that it finds better solutions than naive worst-case linear programming solutions to such problems.