In this paper we discuss improving yield in semiconductor manufacturing using reinforcement learning (RL) to tune a dispatching rule parameter to increase the number of lots that process on high-yield equipment. We consider a dispatching rule with a parameter that controls whether or not the rule allows a lot to process on a lower-yield equipment or waits to allow the lot to possibly process later on a high-yield equipment. In a factory such a parameter would be set periodically, e.g., once a week, but RL allows the parameter to be updated frequently, leading to better factory performance. We also consider a novel measure of on-time delivery where the goal is to have 95% on-time delivery in a set of time intervals, e.g., shifts. We show how a trained RL agent using a graph neural network outperforms the baseline by maintaining on-time delivery while processing significantly more lots on high-yield equipment.

While current systems for autonomous robot navigation can produce safe and efficient motion plans in static environments, they usually generate suboptimal behaviors when multiple robots must navigate together in confined spaces. For example, when two robots meet each other in a narrow hallway, they may either turn around to find an alternative route or collide with each other. This paper presents a new approach to navigation that allows two robots to pass each other in a narrow hallway without colliding, stopping, or waiting. Our approach, Perceptual Hallucination for Hallway Passing (PHHP), learns to synthetically generate virtual obstacles (i.e., perceptual hallucination) to facilitate passing in narrow hallways by multiple robots that utilize otherwise standard autonomous navigation systems. Our experiments on physical robots in a variety of hallways show improved performance compared to multiple baselines.

Despite the advancement of machine learning techniques in recent years, state-of-the-art systems lack robustness to "real world" events, where the input distributions and tasks encountered by the deployed systems will not be limited to the original training context, and systems will instead need to adapt to novel distributions and tasks while deployed. This critical gap may be addressed through the development of "Lifelong Learning" systems that are capable of 1) Continuous Learning, 2) Transfer and Adaptation, and 3) Scalability. Unfortunately, efforts to improve these capabilities are typically treated as distinct areas of research that are assessed independently, without regard to the impact of each separate capability on other aspects of the system. We instead propose a holistic approach, using a suite of metrics and an evaluation framework to assess Lifelong Learning in a principled way that is agnostic to specific domains or system techniques. Through five case studies, we show that this suite of metrics can inform the development of varied and complex Lifelong Learning systems. We highlight how the proposed suite of metrics quantifies performance trade-offs present during Lifelong Learning system development - both the widely discussed Stability-Plasticity dilemma and the newly proposed relationship between Sample Efficient and Robust Learning. Further, we make recommendations for the formulation and use of metrics to guide the continuing development of Lifelong Learning systems and assess their progress in the future.

With robots poised to enter our daily environments, they will not only need to work for people, but also learn from them. An active area of investigation in the robotics, machine learning, and human-robot interaction communities is the design of teachable robots that can learn interactively from human input. To refer to these research efforts, we use the umbrella term Human-Interactive Robot Learning (HIRL). While algorithmic solutions for robots learning from people have been investigated in a variety of ways, HIRL, as a fairly new research area, is still lacking: 1) a formal set of definitions to classify related but distinct research problems or solutions, 2) benchmark tasks, interactions, and metrics to evaluate the performance of HIRL algorithms and interactions, and 3) clear long-term research challenges to be addressed by different communities. Last year we began consolidating the needed definitions and vocabulary to enable fruitful discussions between researchers from these interdisciplinary fields, and identified a preliminary list of long, medium, and short-term research problems for the community to tackle, and existing tools and frameworks that can be leveraged to this end. This workshop will build upon these discussions, focusing on promoting the specification and design of HIRL benchmarks.

With robots poised to enter our daily environments, we conjecture that they will not only need to work for people, but also learn from them. An active area of investigation in the robotics, machine learning, and human-robot interaction communities is the design of teachable robotic agents that can learn interactively from human input. To refer to these research efforts, we use the umbrella term Human-Interactive Robot Learning (HIRL). While algorithmic solutions for robots learning from people have been investigated in a variety of ways, HIRL, as a fairly new research area, is still lacking: 1) a formal set of definitions to classify related but distinct research problems or solutions, 2) benchmark tasks, interactions, and metrics to evaluate the performance of HIRL algorithms and interactions, and 3) clear long-term research challenges to be addressed by different communities. The main goal of this workshop will be to consolidate relevant recent work falling under the HIRL umbrella into a coherent set of long, medium, and short-term research problems, and identify the most pressing future research goals in this area. As HIRL is a developing research area, this workshop is an opportunity to break the existing boundaries between relevant research communities by developing and sharing a diverse set of benchmark tasks and metrics for HIRL, inspired by other fields including neuroscience, biology, and ethics research.

Queue-time constraints (QTC) define a limit on the time that a lot can wait between two process steps in its flow. In semiconductor manufacturing, lots that exceed that time limit experience yield loss, need rework, or get scraped. QTCs are difficult to schedule, since a lot needs to wait to be released to the first process step until there is available capacity to process the final step. However, exactly calculating if there is enough capacity is computationally expensive. In this work we propose a deep Reinforcement Learning (RL) method to manage releasing lots into the queue time constraint. We analyze the performance of our RL method and compare it to seven baseline solutions. Our empirical evaluation shows that the RL method outperforms the baselines in five performance metrics including the number of queue-time violations and makespan, while requiring negligible online compute time.

Chore division is a class of fair division problems in which some undesirable "resource" must be shared among a set of participants, with each participant wanting to get as little as possible. Typically the set of participants is fixed and known at the outset. This paper introduces a novel variant, called sequential online chore division (SOCD), in which participants arrive and depart online, while the chore is being performed: both the total number of participants and their arrival/departure times are initially unknown. In SOCD, exactly one agent must be performing the chore at any give time (e.g. keeping lookout), and switching the performer incurs a cost. In this paper, we propose and analyze three mechanisms for SOCD: one centralized mechanism using side payments, and two distributed ones that seek to balance the participants' loads. Analysis and results are presented in a domain motivated by autonomous vehicle convoy formation, where the chore is leading the convoy so that all followers can enjoy reduced wind resistance.

Creating robots that can perform general-purpose service tasks in a human-populated environment has been a longstanding grand challenge for AI and Robotics research. One particularly valuable skill that is relevant to a wide variety of tasks is the ability to locate and retrieve objects upon request. This paper models this skill as a Scavenger Hunt (SH) game, which we formulate as a variation of the NP-hard stochastic traveling purchaser problem. In this problem, the goal is to find a set of objects as quickly as possible, given probability distributions of where they may be found. We investigate the performance of several solution algorithms for the SH problem, both in simulation and on a real mobile robot. We use Reinforcement Learning (RL) to train an agent to plan a minimal cost path, and show that the RL agent can outperform a range of heuristic algorithms, achieving near optimal performance. In order to stimulate research on this problem, we introduce a publicly available software stack and associated website that enable users to upload scavenger hunts which robots can download, perform, and learn from to continually improve their performance on future hunts.

In this paper, we study the problem of wireless sensor network (WSN) maintenance using a team of physical autonomous mobile agents. The agents are deployed in the area of the WSN in such a way that would minimize the time it takes them to reach a failed sensor and repair it. The team must constantly optimize its collective deployment to account for occupied agents. The objective is to define the optimal deployment and task allocation strategy, that minimize the solution cost. The solution cost is a linear combination of the weighted sensors' downtime, the agents' traveling distance, and penalties incurred due to unrepaired sensors within a certain time limit. Our proposed solution algorithms are inspired by research in the field of computational geometry and the design of our algorithms is based on state of the art approximation algorithms for the classical problem of facility location. We empirically compare and analyze the performance of several proposed algorithms. The sensitivity of the algorithms' performance to the following parameters is analyzed: agents to sensors ratio, sensors' sparsity, frequency and distribution of failures, repair duration, repair capacity, and communication limitations. Our results demonstrate that: (i) cooperation enhances the team's performance by orders of magnitude, (ii) k-Median based deployment algorithm provides up to 30% improvement in downtime, (iii) k-Center based deployment incurs 10% fewest penalties, and (iv) k-Centroid based deployment is most efficient in terms of minimizing the overall costs, with up to 21% lower cost than the next best algorithm.

Traffic congestion is a major challenge in modern urban settings. The industry-wide development of autonomous and automated vehicles (AVs) motivates the question of how can AVs contribute to congestion reduction. Past research has shown that in small scale mixed traffic scenarios with both AVs and human-driven vehicles, a small fraction of AVs executing a controlled multiagent driving policy can mitigate congestion. In this paper, we scale up existing approaches and develop new multiagent driving policies for AVs in scenarios with greater complexity. We start by showing that a congestion metric used by past research is manipulable in open road network scenarios where vehicles dynamically join and leave the road. We then propose using a different metric that is robust to manipulation and reflects open network traffic efficiency. Next, we propose a modular transfer reinforcement learning approach, and use it to scale up a multiagent driving policy to outperform human-like traffic and existing approaches in a simulated realistic scenario, which is an order of magnitude larger than past scenarios (hundreds instead of tens of vehicles). Additionally, our modular transfer learning approach saves up to 80% of the training time in our experiments, by focusing its data collection on key locations in the network. Finally, we show for the first time a distributed multiagent policy that improves congestion over human-driven traffic. The distributed approach is more realistic and practical, as it relies solely on existing sensing and actuation capabilities, and does not require adding new communication infrastructure.

In this work, we present methods for using human-robot dialog to improve language understanding for a mobile robot agent. The agent parses natural language to underlying semantic meanings and uses robotic sensors to create multi-modal models of perceptual concepts like red and heavy. The agent can be used for showing navigation routes, delivering objects to people, and relocating objects from one location to another. We use dialog clari_cation questions both to understand commands and to generate additional parsing training data. The agent employs opportunistic active learning to select questions about how words relate to objects, improving its understanding of perceptual concepts. We evaluated this agent on Amazon Mechanical Turk. After training on data induced from conversations, the agent reduced the number of dialog questions it asked while receiving higher usability ratings. Additionally, we demonstrated the agent on a robotic platform, where it learned new perceptual concepts on the y while completing a real-world task.

Traffic congestion is a major challenge in modern urban settings. The industry-wide development of autonomous and automated vehicles (AVs) motivates the question of how can AVs contribute to congestion reduction. Past research has shown that in small scale mixed traffic scenarios with both AVs and human-driven vehicles, a small fraction of AVs executing a controlled multiagent driving policy can mitigate congestion. In this paper, we scale up existing approaches and develop new multiagent driving policies for AVs in scenarios with greater complexity. We start by showing that a congestion metric used by past research is manipulable in open road network scenarios where vehicles dynamically join and leave the road. We then propose using a different metric that is robust to manipulation and reflects open network traffic efficiency. Next, we propose a modular transfer learning approach and use it to scale up the multiagent driving policy to a realistic simulated scenario that is an order of magnitude larger than past scenarios (hundreds rather than tens of vehicles). Our experimental study shows that the resulting policy improves traffic efficiency over human-driven traffic in a large open network, where existing approaches fail to do so. Another key advantage of our modular transfer learning approach is that it avoids collecting samples from entire network, which saves up to 80% of training and data collection time in our experiments.

As multi-robot applications become more prevalent, it becomes necessary to develop navigation systems which allow autonomous mobile robots to efficiently and safely pass each other in confined spaces. Existing navigation systems, such as the widely used ROS Navigation Stack, usually produce safe, collision free paths in static environments. However, these systems are not perfect, and when multiple mobile robots simultaneously navigate in narrow spaces, collisions and turnarounds are not uncommon. Fine-tuning and enhancing such navigation stacks is not as simple as it looks since they are made up of multiple layers of code, and there exists a tradeoff between optimizing for efficiency, i.e. minimizing time to destination (TTD) vs. optimizing for safety, i.e. minimizing collisions, with each objective leading to a different combination of parameter values. In this paper we develop a methodology to improve existing navigation stacks with regards to both objectives, without tuning their parameters, while preserving their inherent safety control properties. Our proposed approach is a decentralized learning-based approach that is geared toward real world robotic deployment, by requiring little computing resources. It is agnostic of the underlying navigation stack and can adapt to different types of environmental layouts (i.e., hallway structures).

In ad hoc teamwork, multiple agents need to collaborate without having knowledge about their teammates or their plans a priori. A common assumption in this research area is that the agents cannot communicate. However, just as two random people may speak the same language, autonomous teammates may also happen to share a communication protocol. This paper considers how such a shared protocol can be leveraged, introducing a means to reason about Communication in Ad Hoc Teamwork (CAT). The goal of this work is enabling improved ad hoc teamwork by judiciously leveraging the ability of the team to communicate. We situate our study within a novel CAT scenario, involving tasks with multiple steps, where teammates' plans are unveiled over time. In this context, the paper proposes methods to reason about the timing and value of communication and introduces an algorithm for an ad hoc agent to leverage these methods. Finally, we introduces a new multiagent domain, the tool fetching domain, and we study how varying this domain's properties affects the usefulness of communication. Empirical results show the benefits of explicit reasoning about communication content and timing in ad hoc teamwork.

In this paper, we study the problem of data gathering in ad-hoc sensor networks using a mobile entity called mule. The mule traverses the children of failed sensors, to prevent loss of data. Our objective is to define the optimal communication tree, and the mule's placement such that the mule's overall traveling distance is minimized. We explore this problem in several network topologies including: unit disc graph on a line (UDL), general unit disc graph (UDG), and a complete graph with failing probabilities on the nodes (CGFP). We provide an optimal solution for the UDL problem and three approximation algorithms for the UDG problem. For the CGFP problem we outline the two possible structures of an optimal solution and provide near optimal approximation algorithms.

This paper defines a novel formulation of chore division, called S equential O nline C hore D ivision (SOCD), in which participants arrive and depart online, while the chore is being performed. In SOCD, there exists some uncertainty both regarding the total number of participants and their arrival/departure times. Moreover, only one agent can perform the chore at any given moment, and switching the performer incurs a cost. This novel variant of chore division can model real world problems such as the autonomous vehicle convoy formation problem, which has significant social implications. Autonomous vehicles are said to form a convoy when vehicles headed in the same direction follow each other in close proximity. This behavior has been proven to save energy, due to the reduction in aerodynamic drag. Empirical evaluations estimate that a follower can save over $10%$ of its fuel consumption \citelammert2018influences. However, since the leader sees little or no such gains, choosing the leader of such a convoy raises issues of fairness, efficiency, and strategyproofness. Solving these issues is challenging since vehicles can dynamically join and leave the convoy. To address this problem, we propose three mechanisms for fair chore division. The first mechanism is centralized and uses side payments while the other two are distributed and seek to balance the participants' loads. We show that the payment-transfer mechanism, which requires a centralized server, results in optimal fairness and efficiency. For the cases where a central server is not available, we show that the repeated-game mechanism produces allocations which are efficiently-optimal and fair in expectation. For the single-game case, we first prove that optimal fairness is impossible to guarantee. We then show that our proposed single-game mechanism, which offers minimal efficiency loss, achieves ex-ante proportionality.

Multi-robot planning (mrp) aims at computing plans, each in the form of a sequence of actions, for a team of robots to achieve their individual goals, while minimizing overall cost. Solving mrp problems requires modeling limited domain resources (e.g., corridors that allow at most one robot at a time), and the possibility of action synergy (e.g., multiple robots going through a door after a single door-opening action). Optimally solving mrp problems is hard as it is a generalization of the single agent planning domain which is known to be NP-hard, and frequently requires considering the states of all the robots, resulting in an exponentially growing joint state and action space. In many mrp domains, robots encounter situations where they have conflicting needs for constrained resources, or where they can take advantage of what each other is doing to form synergies. In this article, we formulate the problem of multi-robot planning with conflicts and synergies (mrpcs), and develop a multi-robot planning framework, called iterative inter-dependent planning (iidp), for representing and solving mrpcs problems. Within the iidp framework, we develop the algorithms of increasing dependency and best alternative which exhibit different trade-offs between plan quality and computational efficiency. Extensive experiments covering the suggested algorithms have been performed using both an abstract-domain simulator, where we can automatically generate a variety of domain configurations, and a practical mrpcs instantiation that focuses on multi-robot navigation. In the navigation domain, we model plan costs with temporal uncertainty, and present a novel shifted-Poisson distribution for accumulating temporal uncertainty over actions. In comparison to baseline approaches, our algorithms produce significant reductions in overall plan cost, while avoiding search in the joint state space. In addition, we present a complete demonstration of the implementation of the model on a team of real robots.

Natural language understanding for robotics can require substantial domain-and platform-specific engineering.For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties.One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamicallycontinually learning new language constructions and perceptual concepts.In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding.We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world.

Efficiently guiding humans in indoor environments is a challenging open problem. Due to recent advances in mobile robotics and natural language processing, it has recently become possible to consider doing so with the help of mobile, verbally communicating robots. In the past, stationary verbal robots have been used for this purpose at Microsoft Research, and mobile non-verbal robots have been used at UT Austin in their multi-robot human guidance system. This paper extends that mobile multi-robot human guidance research by adding the element of natural language instructions, which are dynamically generated based on the robots’ path planner, and by implementing and testing the system on real robots. Generating natural language instructions from the robots’ plan opens up a variety of optimization opportunities such as deciding where to place the robots, where to lead humans, and where to verbally instruct them. We present experimental results of the full multi-robot human guidance system and show that it is more effective than two baseline systems: one which only provides humans with verbal instructions, and another which only uses a single robot to lead users to their destinations.

In this paper, we study the problem of wireless sensor network (WSN) maintenance using a team of collaborative autonomous mobile agents. The agents are deployed in the area of the WSN in such a way that would minimize the time it takes them to reach a failed sensor and repair it. The team must constantly optimize its collective deployment to account for occupied agents. The objective is to define the optimal deployment and task allocation strategy, that minimize the solution cost. The solution cost is a linear combination of the weighted sensors’ downtime, the agents’ traveling distance and penalties incurred due to unrepaired sensors within a certain time limit. A constrained variation of the problem where agents are subject to capacity constraints is also considered. Our solutions are inspired by research in the field of computational geometry and the design of our algorithms is based on state of the art approximation algorithms for the classical problem of facility location. We empirically compare and analyze the performance of several proposed algorithms. The sensitivity of the algorithms’ performance to the following parameters is analyzed: agents to sensors ratio, sensors’ sparsity, frequency of failures and repair duration.