Detecting and tracking people is becoming more important in robotic applications because of the increasing demand for collaborative work in which people interact closely with and in the same workspace as robots. New safety standards allow people to work next to robots, but require that they be protected from harm while they do so. Sensors that detect and track people are a natural way of implementing the necessary safety monitoring, and have the added advantage that the information about where the people are and where they are going can be fed back into the application and used to give the robot greater situational awareness for performing tasks. The results should help users determine if such a system will provide sufficient protection for people to be able to work safely in collaborative applications with industrial robots.

Industrial robots can perform motion with sub-millimeter repeatability when programmed using the teach-and-playback method. While effective, this method requires significant up-front time, tying up the robot and a person during the teaching phase. Off-line programming can be used to generate robot programs, but the accuracy of this method is poor unless supplemented with good calibration to remove systematic errors, feed-forward models to anticipate robot response to loads, and sensing to compensate for unmodeled errors. These increase the complexity and up-front cost of the system, but the payback in the reduction of recurring teach programming time can be worth the effort. This payback especially benefits small-batch, short-turnaround applications typical of small-to-medium enterprises, who need the agility afforded by off-line application development to be competitive against low-cost manual labor. To fully benefit from this agile application tasking model, a common representation of tasks should be used that is understood by all of the resources required for the job: robots, tooling, sensors, and people. This paper describes an information model, the Canonical Robot Command Language (CRCL), which provides a high-level description of robot tasks and associated control and status information.

Detecting and tracking people is becoming more important in robotic applications because of the increasing demand for collaborative work in which people interact closely with and in the same workspace as robots.New safety standards allow people to work next to robots but require that they be protected from harm while they do so.Sensors that detect and track people are a natural way of implementing the necessary safety monitoring and have the added advantage that the information about where the people are and where they are going can be fed back into the application and used to give the robot greater situational awareness.This report describes work on defining and testing performance evaluation measures that provide quantitative information about how well a human detection and tracking system performs.The results should help users determine if such a system will provide sufficient protection for people to be able to work safely in collaborative applications with industrial robots.

We have been researching three dimensional (3D) ground-truth systems for performance evaluation of vision and perception systems in the fields of smart manufacturing and robot safety. In this paper we first present an overview of different systems that have been used to provide ground-truth (GT) measurements and then we discuss the advantages of physically-sensed ground-truth systems for our applications. Then we discuss in detail the three ground- truth systems that we have used in our experiments: ultra wide-band, indoor GPS, and a camera-based motion capture system. Finally, we discuss three different perception-evaluation experiments where we have used these GT systems.

Collaborative robots are used in close proximity to humans to perform a variety of tasks, while more traditional industrial robots are required to be stopped whenever a human enters their work-volumes. Instead of relying on physical barriers or merely detecting when someone enters the area, the collaborative system must monitor the position of every person who enters the work space in time for the robot to react. The TC 184/SC 2/WG 3 Industrial Safety group within the International Organization for Standard(ISO) is developing the standards to help ensure collaborative robots operate safely. Collaborative robots require sophisticated sensing technologies that must handle dynamic interactions between the robot and the human. One potential safety risk is the occlusion of a safety sensor's field of view due to placement of objects or the movement of people in front of a safety sensor. In this situation the robot could shut down as soon as even a single sensor was partially occluded. Unfortunately this could greatly diminish the extent to which the robot could work collaboratively. In this paper we examine how a human tracking system using multiple laser line scanners [3]was adapted to work with a robot Speed and Separation Monitoring (SSM) safety system and further modified to include occlusion monitoring.

The National Institute of Standards and Technology‟s Intelligent Systems Division has been researching several areas leading to safe control of manufacturing vehicles to improve automated guided vehicle (AGV) safety standards. The research areas include:  AGV safety and control based on advanced two-dimensional (2D) sensors that detect moving standard test pieces representing humans;  Ability of advanced 3D imaging sensors, when mounted to an AGV or forklift, to detect stationary or moving objects and test pieces on the ground or hanging over the work area; and  Manned forklift safety based on advanced 3D imaging sensors that detect visible and non-visible regions for forklift operators. Experiments and results in the above areas are presented in this paper. The experimental results will be used to develop and recommend standard test methods, some of which are proposed in this paper, and to improve the standard stopping distance exception language and operator blind spot language in AGV standards.

The National Institute of Standards and Technology's Intelligent Systems Division has been researching automated guided vehicle (AGV) control based on advanced two-dimensional (2D) imaging sensors that detect dynamic, standard test pieces representing humans towards improving AGV safety standards. Experiments and results are presented in this paper showing the measurement of dynamic standard test pieces from an automated guided vehicle as compared to ground truth. The experimental results will be used to develop standard test methods and to recommend improved standard stopping distance exception language to AGV standards.

The National Institute of Standards and Technology's Intelligent Systems Division has been researching advanced three-dimensional (3D) imaging sensors and their use in manufacturing towards improving forklift safety. Experiments are presented in this paper and that show how the sensors can augment a forklift operator's perception of obstacles nearby. Interoperability of the obstacle/pedestrian detection information from these sensors to the facility or other forklifts for broader alerts is also possible.

The test bed for the Measurement Science Program at the National Institute of Standards and Technology (NIST) includes a robot arm on a rail, conveyers, mannequins and a custom designed automated guided vehicle (AGV). The NIST AGV includes an onboard personal computer and control software, safety sensors, and an absolute positioning sensor system among other useful and research tools, sensors and equipment. A NIST-developed, smart diagnostic tool was used to design, adjust, and monitor vehicle parameters and control algorithms to enable robust autonomous vehicle control. The NIST Diagnostics Tool and its application to the NIST AGV are presented in this paper.

This paper describes the role of the Primitive Mobility (PrimMob) module within the Mobility Open Architecture Simulation and Tools(MOAST) environment. Descriptions are given of several alternative implementations of the module motivated by a desire to make MOAST more usable for an industrial Automated Guided Vehicle(AGV). A series of performance metrics is described as a series of tests that allow those performance metrics to be determined. Tools added to MOAST in 2009-2010 that greatly ease recording and visualization of these performance metrics are also described.

This paper will describe a flexible and inexpensive method of obtaining ground truth for the evaluation of Human Tracking systems. It is expected to be appropriate for evaluating systems used to allow robots and/or autonomous vehicles to operate safely around humans. It is currently focused on tracking people as they stand still or walk. It relies on multiple Laser Measurement Sensors(LMS) also called laser line scanners. The LMS's are mounted to scan in a horizontal plane. A method for quickly calibrating the relative position and orientation of each of the sensors to each other is described. A basic human tracking algorithm using the LMS's is described along with how the algorithm can be combined with a priori knowledge of the walkers intended path during the test. A graphical user interface(GUI) displays both the data obtained directly from the LMS and the output of the tracking algorithm. The GUI allows the user to verify and adjust the tracking algorithm without needing to annotate every frame, and therefore at a lower cost than systems that require extensive annotation. Tests were performed with people walking or running though several patterns, while data was simultaniously recorded by a more expensive system require individual receivers on each participant for comparison.

Abstract : Abstract?The Defense Applied Research Projects Agency (DARPA) Learning Applied to Ground Vehicles (LAGR) program aims to develop algorithms for autonomous vehicle navigation that learn how to operate in complex terrain. For the LAGR program, The National Institute of Standards and Technology (NIST) has embedded learning into a control system architecture called 4D/RCS to enable the small robot used in the program to learn to navigate through a range of terrain types. This paper describes performance evaluation experiments on one of the algorithms developed under the program to learn terrain traversability. The algorithm uses color and texture to build models describing regions of terrain seen by the vehicle?s stereo cameras. Range measurements from stereo are used to assign traversability measures to the regions. The assumption is made that regions that look alike have similar traversability. Thus, regions that match one of the models inherit the traversability stored in the model. This allows all areas of images seen by the vehicle to be classified, and enables a path planner to determine a traversable path to the goal. The algorithm is evaluated by comparison with ground truth generated by a human observer. A graphical user interface (GUI) was developed that displays an image and randomly generates a point to be classified. The human assigns a traversability label to the point, and the learning algorithm associates its own label with the point. When a large number of such points have been labeled across a sequence of images, the performance of the learning algorithm is determined in terms of error rates. The learning algorithm is outlined in the paper, and results of performance evaluation are described.

In this paper, we describe the process by which we collected sensor data for the evaluation of 3D LIDAR (Light Detection and Ranging). Data were also collected simultaneously from SONAR (Sound Navigation and Ranging) sensors, navigation systems, 2D Laser Measurement Sensor (LMS) and a color camera. We describe software developed to perform data collection and allow for evaluation of the data both offline and in real-time during the data collection and briefly cover the experiments themselves where various obstacles were placed in front of a moving vehicle and results were recorded as to whether the obstacle was detected or not.

In this paper, we describe the current 2D (two dimensional) sensor used for industrial vehicles and ideal sensor configurations for mounting 3D imagers on manufacturing vehicles in an attempt to make them safer. In a search for the ideal sensor configuration, three experiments were performed using an advanced 3D imager and a color camera. The experiments are intended to be useful to the standards community and manned and unmanned forklift and automated guided vehicle industries. The imager that was used was a 3D Flash LIDAR (Light Detection and Ranging) camera with 7.5 m range and rapid detection. It was selected because it shows promise for use on forklifts and other industrial vehicles. Experiments included: 1) detection of standard sized obstacles, 2) detection of obstacles with highly reflective surfaces within detection range, and 3) detection of forklift tines above the floor. We briefly describe these experiments and reference their detailed reports.

The National Institute of Standards and Technology (NIST) Intelligent Control of Mobility Systems (ICMS) Program provides architectures and interface standards, performance test methods and data, and infrastructure technology needed by the U.S. manufacturing industry and government agencies in developing and applying intelligent control technology to mobility systems to reduce cost, improve safety, and save lives. The ICMS Program is made up of several areas including: defense, transportation, and industry projects, among others. Each of these projects provides unique capabilities that foster technology transfer across mobility projects and to outside government, industry and academia for use on a variety of applications. A common theme among these projects is autonomy and the Four Dimensional (3D + time)/Real-time Control System (4D/RCS) standard control architecture for intelligent systems that has been applied to these projects. This chapter will briefly describe recent project advances within the ICMS Program including: goals, background accomplishments, current capabilities, and technology transfer that has or is planned to occur. Several projects within the ICMS Program have developed the 4D/RCS into a modular architecture for intelligent mobility systems, including: an Army Research Laboratory (ARL) Project currently studying onroad autonomous vehicle control, a Defense Advanced Research Project Agency (DARPA) Learning Applied to Ground Robots (LAGR) Project studying learning within the 4D/RCS architecture with road following application, and an Intelligent Systems Ontology project that develops the description of intelligent vehicle behaviors. Within the standards and performance measurements area of the ICMS program, a Transportation Project is studying components of intelligent mobility systems that are finding their way into commercial crash warning systems (CWS). In addition, the ALFUS (Autonomy Levels For Unmanned Systems) project determines the needs for metrics and standard definitions for autonomy levels of unmanned systems. And a JAUS (Joint Architecture for Unmanned Systems) project is working to set a standard for interoperability between components of unmanned robotic vehicle systems. Testbeds and frameworks underway at NIST include the PRIDE (Prediction in Dynamic Environments) framework to provide probabilistic predictions of a moving object's future position to an autonomous vehicle's planning system, as well as the USARSim/MOAST (Urban Search and Rescue Simulation/Mobility Open Architecture Simulation and Tools) framework that is being developed to provide a comprehensive set of open source tools for the development and evaluation of autonomous agent systems. A NIST Industrial Autonomous Vehicles (IAV) Project provides technology transfer from the defense and transportation projects directly to industry through collaborations with automated guided vehicles manufacturers by researching 4D/RCS control applications to automated guided vehicles inside facilities. These projects are each briefly described in this Chapter followed by Conclusions and continuing work.

The Intelligent Systems Division at the National Institute of Standards and Technology (NIST) has participated in the Defense Advanced Research Project Agency (DARPA) Learning Applied to Ground Robots (LAGR) project for the past 2 1/2 years. In Phase 2 of the LAGR program, NIST was asked to provide a common operator control unit (OCU) color scheme for all LAGR teams to use. The color scheme simplifies the task of LAGR's evaluation team by providing a straightforward way to compare the performance of each of the teams using the different OCUs. During Phase 1, LAGR performers applied their own standards to the OCU color scheme and DARPA and other performers had a very difficult experience evaluating what the robot was computing based on stereo image, instrumented bumper, and inertial data. NIST developed the color scheme based on real-world conventions and on the desire to accommodate as much of the teams' existing color schemes as possible. For example, typically red lights mean stop and green lights mean go for automobiles. This scheme was adopted by coloring obstacles red and traversable ground green in the new common color scheme. Red, green, blue (RGB) colors were produced for a variety of necessary parameters including: unknown regions, lethals, bumper hits, road/path, planned and traversed paths, goal, and waypoints. Also, vehicle modes were expressed such as: Normal Control, Aggressive, Backing, Stopped, and Manual modes. The paper discusses the color scheme for ground robots developed for the LAGR Program.