Artificial intelligence (AI) has developed significantly in recent years. Its increased application in the industrial and domestic worlds raises questions about how it complements human intelligence. It seems only possible to evaluate this complementarity task by task or capability by capability. This chapter proposes a method and criteria (nature of the evaluation task, application area, level of difficulty, etc.) for systematising tasks on which AI and robotics systems have been evaluated in the past. This will allow the extraction of areas already covered and those yet to be evaluated. This method is applied to evaluation campaigns by the National Institute of Standards and Technology in the United States and the French Laboratoire National de Métrologie et d'Essais over the last decades. The paper concludes with a proposal for next steps to complete the mapping based on expert judgement.

Provides guidance for small and medium sized manufacturers to identify the workcell best served by cobot integration, starting with basic and quick cobot integration methods, and moving to more time-consuming but more accurate cobot integration methods.

This article presents a set of performance metrics, test methods, and associated artifacts to help progress the development and deployment of robotic assembly systems. The designs for three task board artifacts that replicate small part insertion and fastening operations such as threading, snap fitting, and meshing with standard screws, nuts, washers, gears, electrical connectors, belt drives, and wiring are presented. To support the evaluation of robotic assembly and disassembly operations, benchmarking protocols and performance metrics are presented that leverage these task boards. Finally, robot competitions are discussed as use cases for these task boards.

This letter describes a set of metrics and supporting benchmarking protocols for determining the performance characteristics of robot end-effectors. In the short-term, these tools are proving useful as a common ground for assessing and comparing end-effectors. The long-term goal is a standard framework for providing technical specifications for robotic end-effectors to help pair technologies to application spaces. This letter presents a subset of the metrics - grasp strength, grasp cycle time, finger strength, and finger repeatability - with accompanying measurement techniques and supporting test artifacts. The application of these metrics and protocols is demonstrated using example implementations to characterize a variety of robot end-effectors, with example data sets and test designs provided for downloading.

Rigorous experiments enabling reproducibility are needed to advance the rapidly growing field of robotics more efficiently.

A-UGV has been defined by ASTM Committee F45 as an “Automatic, Automated, or Autonomous vehicle that operates while in contact with the ground without a human operator”. However, what do the levels actually mean to manufacturers, users, or especially potential users? This paper defines, and in many cases provides examples of, recommended autonomy levels for all three automatic, automated, and autonomous unmanned ground vehicles.

Increasing the flexibility and general-purpose applicability of robots are longstanding goals for many application domains.Several avenues of research addressing these goals include robotic hand mechatronic design, sensing, and control.Inspired by nature, these end-effectors hold potential for allowing robots to grasp and manipulate a broader range of objects without requiring customized end-of-arm tooling or grippers.With the rapidlygrowing number and diversity of robot hands, there is a need to quantify their individual capabilities and characteristics under a unified framework.Terminology and associated definitions are needed in order to support this performance measurement framework.Terminology that is agreed-upon by the stakeholders will facilitate communication among individuals involved in the research, design, and integration of robotic hands through a common and consistent lexicon.Furthermore, common terminology will improve communication with end-users, instilling effective matching of technology capabilities with end-user requirements.This publication proposes a draft set of definitions as a means of stimulating discussion among researchers, developers, integrators, and end-users of robotic hands.This draft document may one day provide input to a consensus standard on terminology for robotic hands.Resources, in addition to existing standards, such as industry terminology lists may be leveraged by contributors to this document with appropriate reference.

Increasing the flexibility and general-purpose applicability of robots is a longstanding goal.Several avenues of research are addressing these goals, ranging from integration of multiple sensors to allow robots to perceive their surroundings and adapt accordingly, to more sophisticated control algorithms that enable robots to re-plan based on current status, to development of more dexterous means of manipulating objects.As part of the manipulation thrust, there has been a recent increase in the development of robotic hands.Inspired by nature, these end effectors hold potential for allowing robots to pick up and manipulate a broader range of objects, without requiring customized end-of-arm tooling or grippers.With this rapidly-growing number of robot hands with diverse designs, there is a need to capture their individual competencies and characteristics under a unified framework.In addition to knowledge of basic hand characteristics such as the number of fingers, degrees of freedom, and degrees of actuation, performance metrics can provide valuable insight into not only the raw traits of the technology, but also their task and function-level performance capabilities.These measures can then be used to help match capabilities to end-user needs as well as provide researchers and developers insight for improving their hardware and software designs.

The Robot Grasping and Manipulation Competition, held during the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) in Daejeon, South Korea was sponsored by the IEEE Robotic and Automation Society (RAS) Technical Committee (TC) on Robotic Hands Grasping and Manipulation (RHGM) [1]. This competition was the first of a planned series of grasping and manipulation-themed events of increasing difficulty that are intended to spur technological developments and advance test methods and benchmarks so that they can be formalized for use by the community. The coupling of standardized performance testing with robot competitions will promote the use of unbiased evaluation methods to assess how well a robot system performs in a particular application space. A strategy is presented for a series of grasping and manipulation competitions that facilitate objective performance benchmarking of robotic assembly solutions. This strategy is based on test methods that can be used for more rigorous assessments and comparison of systems and components outside of the competition regime. While competitions have proven to be useful mechanisms for assessing the relative performance of robotic systems with measures of success, they often lack a methodical measurement science foundation. Consequently, scientifically sound and statistically significant metrics, measurement, and evaluation methods to quantify performance are missing. Using performance measurement methods in a condensed format will accommodate competition time limits while introducing the methods to the community as tools for benchmarking performance in the developmental and deployment phases of a robot system. The particular evaluation methods presented here are focused on the mechanical assembly process, an application space that is expected to accelerate with the new robot technologies coming to market.

The U.S. National Institute of Standards and Technology (NIST) is dedicated to advancing measurement science, technology, and standards, thus promoting innovation and industry competitiveness. In particular, the Robot Systems for Smart Manufacturing (RSSM) program within NIST’s Engineering Laboratory focuses on advancing measurement science for industrial robot systems, specifically to address issues that may hinder ease of adoption and advances in the performance of robotic technologies. Robotic systems are an essential tool for strengthening manufacturing competitiveness in our rapidly changing global economy.

Manufacturing robotics is moving towards human-robot collaboration with light duty robots being used side by side with workers. Similarly, exoskeletons that are both passive (spring and counterbalance forces) and active (motor forces) are worn by humans and used to move body parts. Exoskeletons are also called ‘wearable robots’ when they are actively controlled using a computer and integrated sensing. Safety standards now allow, through risk assessment, both manufacturing and wearable robots to be used. However, performance standards for both systems are still lacking. Ongoing research to develop standard test methods to assess the performance of manufacturing robots and emergency response robots can inspire similar test methods for exoskeletons. This paper describes recent research on performance standards for manufacturing robots as well as search and rescue robots. It also discusses how the performance of wearable robots could benefit from using the same test methods.

Automated guided vehicles (AGVs) typically have been used for industrial material handling since the 1950s. In the years following, U.S. and European safety standards have been evolving to protect nearby workers. However, no performance standards have been developed for AGV systems. In our view, lessons can be learned for developing such standards from the research and standards associated with mobile robots applied to search and rescue and military applications. Research challenge events, tests and evaluations, and intelligence-level efforts have also occurred that can support industrial AGV developments into higher-level intelligent systems and provide useful standards development criteria for AGV performance test methods. This chapter provides background information referenced from all of these areas to support the need for an AGV performance standard.

Description Get 8 papers that detail perspectives from related standards efforts, industry needs, and cutting-edge research on automatic guided vehicles (AGVs). The authors discuss performance standards and benchmarks that can enable technology transfer from the laboratory to industry. These papers tackle the complexities of multiple AGVs operating in unstructured environments. This book comprises expanded versions of selected papers presented at the Institute of Electrical and Electronic Engineers International Conference on Robotics and Automation (ICRA) workshop. This research sets the stage by reviewing standards development for other mobile robot application domains, such as emergency response, and suggests approaches for tackling performance measurement for intelligent AGVs. The authors discuss examples of performance standards that could be used for vehicle navigation performance and for perception systems. Automatic guided vehicles were one of the earliest applications for mobile robots. The first AGVs were deployed in the 1950s to transport materials in large facilities and warehouses. Mobile robot capabilities have advanced significantly in the past decades. General-purpose sensing, planning, communications, and control are paving the way for an era where the “A” stands for “autonomous.” This evolution in onboard intelligence has greatly expanded the potential scope of applications for AGVs; thus, ASTM has formed a new committee to develop these missing standards for measuring, describing, and characterizing performance for this new breed of AGVs. ASTM’s Committee F45 on Driverless Automatic Guided Industrial Vehicles will standardize nomenclature and definitions of terms, and develop recommended practices, guides, test methods, specifications, and performance standards for AGVs.