We consider multi-robot scenarios where robots ask for operator interventions when facing difficulties. As the number of robots increases, the operator quickly becomes a bottleneck for the system. Queue theory can be effectively used to optimize the scheduling of the robots’ requests. Here we focus on a specific queuing model in which the robots decide whether to join the queue or balk based on a threshold value. Those thresholds are a trade-off between the reward earned by joining the queue and cost of waiting in the queue. Though such queuing models reduce the system’s waiting time, the cost of balking usually is not considered. Our aim is thus to find appropriate balking strategies for a robotic application to reduce the waiting time considering the expected balking costs. We propose using a Q-learning approach to compute balking thresholds and experimentally demonstrate the improvement of team performance compared to previous queuing models.

Team oriented plans have become a popular tool for operators to control teams of autonomous robots to pursue complex objectives in complex environments. Such plans allow an operator to specify high level directives and allow the team to autonomously determine how to implement such directives. However, the operators will often want to interrupt the activities of individual team members to deal with particular situations, such as a danger to a robot that the robot team cannot perceive. Previously, after such interrupts, the operator would usually need to restart the team plan to ensure its success. In this paper, we present an approach to encoding how interrupts can be smoothly handled within a team plan. Building on a team plan formalism that uses Colored Petri Nets, we describe a mechanism that allows a range of interrupts to be handled smoothly, allowing the team to efficiently continue with its task after the operator intervention. We validate the approach with an application of robotic watercraft and show improved overall efficiency. In particular, we consider a situation where several platforms should travel through a set of pre-specified locations, and we identify three specific cases that require the operator to interrupt the plan execution: (i) a boat must be pulled out; (ii) all boats should stop the plan and move to a pre-specified assembly position; (iii) a set of boats must synchronize to traverse a dangerous area one after the other. Our experiments show that the use of our interrupt mechanism decreases the time to complete the plan (up to 48 % reduction) and decreases the operator load (up to 80 % reduction in number of user actions). Moreover, we performed experiments with real robotic platforms to validate the applicability of our mechanism in the actual deployment of robotic watercraft.

Human multi-robot control is a complex process: an operator’s attention constantly shifts among various tasks to maintain the necessary awareness of a given situation, such as exploring environments and locating targets simultaneously. To minimize the laborious monitoring, if a robot can self-reflect its needs for interaction and alert an operator to any abnormalities, the operator’s attention could be appropriately directed to urgent events in a timely manner. This type of human-robot interaction can be viewed as a queuing system, in which an operator acts as a server, and robots’ requests for services are the jobs. This paper presents two experiments that investigated scheduling interventions. In the first experiment, participants performed search and rescue tasks while assisted by an alarmed system: Open-queue, in which all the alarms are displayed at once, or FIFO-queue (first-in-first-out approach), which shows only a single alarm at a time. The second study compared the FIFO and Open conditions from the first study with a shortest-job-first (SJF) queue system. The results show that operator attention can be effectively directed toward interaction with needed robots without degrading performance, which suggest the uses of the queueing disciplines could lead to superior performance while choosing between robot requests becomes more challenging. Additionally, to better understand operators’ intention and resulting behaviors, a Hidden Markov Model (HMM) was applied to investigate human control processes in interactions with multi-robot teams. The HMM results demonstrated fundamental differences among queuing mechanisms, which are difficult to observe through conventional approaches.

Environmental monitoring is one of the main societal challenge of the last century. This is particularly important when considering the increasing population growth and the industrialization of developing countries. Information and Communication Technology (ICT) and robotics can play a significant role in this perspective as they offer invaluable tools to continuously gather relevant information over wide, possibly dangerous areas. In particular autonomous boats hold great promises for monitoring water bodies (lakes, rivers, fishfarms) or providing first aid in floods. However, a key feature for such systems to have a practical impact on society is to provide accurate measures over large areas while being cost effective. In this work, we focus on a water monitoring system, based on Cooperative Robotic Watercrafts (CRW), commercialised by Platypus LLC . The CRW approach is based on the use of low cost, small, robotics platforms, which are able to interact with each other and make use of various artificial intelligence methods to ensure a significant level of autonomy for navigation and information gathering tasks. Figure 1(a) shows a differential drive propeller versions. In addition to a battery based propulsion mechanism, each boat is equipped with an Android OS smartphone, custom electronics board, and sensor payload. The Android smartphone provides communication, GPS, compass, and multi-core processor. The Arduino Mega based electronics board receives commands from the Android phone and interfaces with the propulsion mechanism and sensor payload, as shown in Figure 1(b). The electronics board supports a wide variety of devices including acoustic doppler current profilers and sensors that measure temperature, Dissolved Oxygen (DO), and pH level. A crucial aspect for the CRW is the high level of autonomy of the platforms, which can be controlled by few human operators that provide high level instructions to the system. In particular, Figure 1(c) shows the Graphical User Interface that the human operators can use to control the robotic platforms. Boats behaviors are expressed in the form of high level plan encoded as Colored Petri Nets [1]. In this example, the human operator activated an explore area plan providing the shape of the area that must be explored. The platforms will then automatically divide the area in different sections and execute a pre-specified strategy to monitor the area. Figure 1(d) shows an example of the standard lawnmower strategy employed by the boats. Notice that such strategy is decided off-line, i.e., before the system acquires any readings, and as such it does not depend on the data that the platforms are collecting.

Team oriented plans have become a popular tool for operators to control teams of autonomous agents (or robots) to pursue complex objectives in complex environments. Such plans allow an operator to specify high level directives and allow the team to autonomously determine how to implement such directives. However, the operators will often want to interrupt the activities of individual team members to deal with particular situations, such as a danger to a robot that the robot team cannot perceive. Previously, after such interrupt, the operator would usually need to restart the team plan to ensure its success. In this paper, we present an approach to encoding how unexpected interrupts can be smoothly handled within a team plan. Building on a team plan formalism that uses Colored Petri Net, we describe a mechanism that allows a range of interrupts to be handled smoothly, allowing the team to efficiently continue with its task, after the operator intervention. We validate the approach with an application of robotic watercraft and show improved overall efficiency. In particular, our interrupt mechanism decreases the time to complete the mission (up to 48% reduction) and decreases the operator load (up to 80% reduction in number of user actions).

This paper presents a team plan specification language that combines work in the creation of generic team plans and design of intelligent interfaces. Two key motivations for developing the language are (1) to combine inter-agent cooperation and operator interaction of complex behaviors into a single plan, and (2) to separate plan design and UI design such that they are created by application domain experts and human interaction experts, respectively. The presented result is a generic language for multi-robot plans that defines tasks to be performed, operator interactions for maintaining situational awareness, and mixed initiative actions to react to operator workload.

The proliferation of unmanned aerial vehicles (UAVs) in civil and military domains has spurred increasingly complex automation design for augmenting operator abilities, reducing workload, and increasing mission effectiveness. We describe the Adaptive Interface Management System (AIMS), an intelligent adaptive delegation interface for controlling and monitoring multiple unmanned vehicles, with a mixed-initiative team model language. A study was conducted to assess understanding of this model language and whether participants exhibited calibrated trust in the intelligent automation. Results showed that operators had accurate memory for role responsibility and were well calibrated to the automation. Adaptive automation design approaches like the one described in this paper can be useful to create mixedinitiative human-robot teams.

The proliferation of unmanned aerial vehicles (UAVs) in civil and military domains has spurred increasingly complex automation design for augmenting operator abilities, reducing workload, and increasing mission effectiveness. We describe the Adaptive Interface Management System (AIMS), an intelligent adaptive delegation interface for controlling and monitoring multiple unmanned vehicles, with a mixed-initiative team model language. A study was conducted to assess understanding of this model language and whether participants exhibited calibrated trust in the intelligent automation. Results showed that operators had accurate memory for role responsibility and were well calibrated to the automation. Adaptive automation design approaches like the one described in this paper can be useful to create mixedinitiative human-robot teams.

Synchronous video has long been the preferred mode for controlling remote robots with other modes such as asynchronous control only used when unavoidable as in the case of interplanetary robotics. We identify two basic problems for controlling multiple robots using synchronous displays: operator overload and information fusion. Synchronous displays from multiple robots can easily overwhelm an operator who must search video for targets. If targets are plentiful, the operator will likely miss targets that enter and leave unattended views while dealing with others that were noticed. The related fusion problem arises because robots' multiple fields of view may overlap forcing the operator to reconcile different views from different perspectives and form an awareness of the environment by piecing them together. We have conducted a series of experiments investigating the suitability of asynchronous displays for multi-UV search. Our first experiments involved static panoramas in which operators selected locations at which robots halted and panned their camera to capture a record of what could be seen from that location. A subsequent experiment investigated the hypothesis that the relative performance of the panoramic display would improve as the number of robots was increased causing greater overload and fusion problems. In a subsequent Image Queue system we used automated path planning and also automated the selection of imagery for presentation by choosing a greedy selection of non-overlapping views. A fourth set of experiments used the SUAVE display, an asynchronous variant of the picture-in-picture technique for video from multiple UAVs. The panoramic displays which addressed only the overload problem led to performance similar to synchronous video while the Image Queue and SUAVE displays which addressed fusion as well led to improved performance on a number of measures. In this paper we will review our experiences in designing and testing asynchronous displays and discuss challenges to their use including tracking dynamic targets. © 2012 by the American Institute of Aeronautics and Astronautics, Inc.

The present study investigates the effect of imperfect automation in a human multi-robot controlled environment with different principles for scheduling an operator's attention in a foraging task. The experiment compared a SJF-queue (shortest job first) presenting a single alarm at a time with an Open-queue which showed all current alarms. Two levels of automation reliability, high (90%) and low (50%), were examined in the study. Performance for the queue mechanisms was equivalent confirming that operator attention can be effectively directed to improve performance. Additionally, the higher reliability condition raised an operator's success rate for resolving robot failures and assisted the operator in allocating attention to emergent events in a timely manner. Although the more frequent alerts contributed to better performance operators experienced increased levels of workload.

Controlling a team of Unmanned Aerial Vehicles (UAV) requires the operator to perform continuous surveillance and path planning. The operator's situation awareness is likely to degrade as an increasing number of surveillance videos must be viewed and integrated. The Picture-in-Picture display (PiP) provides one solution for integrating multiple UAV camera video by allowing the operator to view the video feed in the context of surrounding terrain. The experimental SUAVE (Simple Unmanned Aerial Vehicle Environment) display extends PiP methods by sampling imagery from the video stream to texture a 3D map of the terrain. The operator can then inspect this imagery using world in miniature (WIM) or fly-through methods. Our previous investigation of the properties and advantages of SUAVE in the context of a search mission with 11 UAVs showed a strong advantage for finding targets. We investigated the effects of constrained versus unconstrained fly through technique to evaluate the performance in such spatially immersive displays. Results indicated no difference between these two motion control techniques.

In this paper, we present an asynchronous display method, coined image queue, which allows operators to search through a large amount of data gathered by autonomous robot teams. We discuss and investigate the advantages of an asynchronous display for foraging tasks with emphasis on Urban Search and Rescue. The image queue approach mines video data to present the operator with a relevant and comprehensive view of the environment in order to identify targets of interest such as injured victims. It fills the gap for comprehensive and scalable displays to obtain a network-centric perspective for UGVs. We compared the image queue to a traditional synchronous display with live video feeds and found that the image queue reduces errors and operator's workload. Furthermore, it disentangles target detection from concurrent system operations and enables a call center approach to target detection. With such an approach we can scale up to very large multi-robot systems gathering huge amounts of data that is then distributed to multiple operators.

Controlling a team of Unmanned Aerial Vehicles (UAV) requires the operator to perform continuous surveillance and path planning. The operator's situation awareness is likely to degrade as an increasing number of surveillance videos must be viewed and integrated. The Picture-in-Picture display (PiP) provides one solution for integrating multiple UAV camera video by allowing the operator to view the video feed in the context of surrounding terrain. The experimental SUAVE (Simple Unmanned Areal Vehicle Environment) display extends PiP methods by sampling imagery from the video stream to texture a 3D map of the terrain. The operator can then inspect this imagery using world in miniature (WIM) or fly-through methods. We investigate the properties and advantages of SUAVE in the context of a search mission with 11 UAVs finding a strong advantage for finding targets While performance is expected to improve with increasing numbers of UAVs we did not find differences in performance between models generated by 11 UAVs and those employing 22 UAVs.

This paper describes the software system supporting the Carnegie Mellon/Univ. of Pittsburgh team of simulated search and rescue robots in the Robocup Rescue 2010 Virtual Robots competition. Building on the Machinetta agent software, robot command and control is decomposed into a hierarchy of subtasks managed by independent agents both on the robot and colocated with human operators. By encapsulating all robot and human operator interactions into interfaces to these agents, the system can perform with a high level of robustness and reusability. As in previous years, the entire code base is portable and platform-independent, running entirely in Java.

In this paper, we discuss and investigate the advantages of an asynchronous display, called image queue, for foraging tasks with emphasis on Urban Search and Rescue. The image queue approach mines video data to present the operator with a relevant and comprehensive view of the environment, which helps the user to identify targets of interest such as injured victims. This approach allows operators to search through a large amount of data gathered by autonomous robot teams, and fills the gap for comprehensive and scalable displays to obtain a network-centric perspective for UGVs. It is found that the image queue reduces errors and operator’s workload comparing with the traditional synchronous display. Furthermore, it disentangles target detection from concurrent system operations and enables a call center approach to target detection. With such an approach, it could scale up to a larger multi-robot systems gathering huge amounts of data with multiple operators.

For an interesting class of emerging applications, a large robot team will need to distributedly allocate many more tasks than there are robots, with dynamically appearing tasks and a limited ability to communicate. The LA-DCOP algorithm can conceptually handle both large-scale problems and multiple tasks per robot, but has key limitations when allocating spatially distributed tasks. In this paper, we extend LA-DCOP with several alternative acceptance rules for robots to determine whether to take on an additional task, given the interaction with the tasks it has already committed to. We show that these acceptance rules dramatically outperform a naive LA-DCOP implementation. In addition, we developed a technique that lets the robots use completely local knowledge to adjust their task acceptance criteria to get the best possible performance at a given communication bandwidth level.

In this paper, we discuss and investigate the advantages of an asynchronous display, called “image queue”, tested for an urban search and rescue foraging task. The image queue approach mines video data to present the operator with a relevant and comprehensive view of the environment by selecting a small number of images that together cover large portions of the area searched. This asynchronous approach allows operators to search through a large amount of data gathered by autonomous robot teams, and allows comprehensive and scalable displays to obtain a network-centric perspective for unmanned ground vehicles (UGVs). In the reported experiment automatic target recognition (ATR) was used to augment utilities based on visual coverage in selecting imagery for presentation to the operator. In the cued condition a box was drawn in the region in which a possible target was detected. In the no-cue condition no box was drawn although the target detection probability continued to play a role in the selection of imagery. We found that operators using the image queue displays missed fewer victims and relied on teleoperation less often than those using streaming video. Image queue users in the no-cue condition did better in avoiding false alarms and reported lower workload than those in the cued condition.

This paper describes the software system supporting the Carnegie Mellon/Univ. of Pittsburgh team of simulated search and rescue robots in the Robocup Rescue 2010 Virtual Robots competition. Buildin ...