The new software bdsim is a Python-based tool for simulating continuous- and/or discrete-time dynamical systems (see “Summary”). Systems are conceptualized in block diagram form but are represented directly in Python code, where each block is an object instance. Bdsim can simulate small-scale textbook systems for pedagogical purposes, but it is also capable of modeling large and complex systems of systems, such as the one in Figure 1. Bdsim is an open source Python 3 package [1] available on the Python Package Index (PyPI) [2].

Advanced robotic manipulation of deformable, volumetric objects remains one of the greatest challenges due to their pliancy, frailness, variability, and uncertainties during interaction. Motivated by these challenges, this article introduces Sashimi-Bot, an autonomous multi-robotic system for advanced manipulation and cutting, specifically the preparation of sashimi. The objects that we manipulate, salmon loins, are natural in origin and vary in size and shape, they are limp and deformable with poorly characterized elastoplastic parameters, while also being slippery and hard to hold. The three robots straighten the loin; grasp and hold the knife; cut with the knife in a slicing motion while cooperatively stabilizing the loin during cutting; and pick up the thin slices from the cutting board or knife blade. Our system combines deep reinforcement learning with in-hand tool shape manipulation, in-hand tool cutting, and feedback of visual and tactile information to achieve robustness to the variabilities inherent in this task. This work represents a milestone in robotic manipulation of deformable, volumetric objects that may inspire and enable a wide range of other real-world applications.

There is growing excitement about the potential of leveraging artificial intelligence (AI) to tackle some of the outstanding barriers to the full deployment of robots in daily lives. However, action and sensing in the physical world pose greater and different challenges for AI than analysing data in isolation and it is important to reflect on which AI approaches are most likely to be successfully applied to robots. Questions to address, among others, are how AI models can be adapted to specific robot designs, tasks and environments. This Perspective offers an assessment of what AI has achieved for robotics since the 1990s and proposes a research roadmap with challenges and promises. These range from keeping up-to-date large datasets, representatives of a diversity of tasks that robots may have to perform, and of environments they may encounter, to designing AI algorithms tailored specifically to robotics problems but generic enough to apply to a wide range of applications and transfer easily to a variety of robotic platforms. For robots to collaborate effectively with humans, they must predict human behaviour without relying on bias-based profiling. Explainability and transparency in AI-driven robot control are essential for building trust, preventing misuse and attributing responsibility in accidents. We close with describing what are, in our view, primary long-term challenges, namely, designing robots capable of lifelong learning, and guaranteeing safe deployment and usage, as well as sustainable development. AI technologies are advancing rapidly, offering new solutions for autonomous robot operation in complex environments. Aude Billard et al. discuss the need to identify and adapt AI technologies for robotics, proposing a research roadmap to address key challenges and opportunities.

Vision is a popular and effective sensor for robotics from which we can derive rich information about the environment: the geometry and semantics of the scene, as well as the age, gender, identity, activity and even emotional state of humans within that scene. This raises important questions about the reach, lifespan, and potential misuse of this information. This paper is a call to action to consider privacy in the context of robotic vision. We propose a specific form privacy preservation in which no images are captured or could be reconstructed by an attacker even with full remote access. We present a set of principles by which such systems can be designed, and through a case study in localisation demonstrate in simulation a specific implementation that delivers an important robotic capability in an inherently privacy-preserving manner. This is a first step, and we hope to inspire future works that expand the range of applications open to sighted robotic systems.

Manipulator kinematics is concerned with the motion of each link within a manipulator without considering mass or force. In this article, which is the first in a two-part tutorial, we provide an introduction to modelling manipulator kinematics using the elementary transform sequence (ETS). Then we formulate the first-order differential kinematics, which leads to the manipulator Jacobian, which is the basis for velocity control and inverse kinematics. We describe essential classical techniques which rely on the manipulator Jacobian before exhibiting some contemporary applications. Part II of this tutorial provides a formulation of second and higher-order differential kinematics, introduces the manipulator Hessian, and illustrates advanced techniques, some of which improve the performance of techniques demonstrated in Part I. We have provided Jupyter Notebooks to accompany each section within this tutorial. The Notebooks are written in Python code and use the Robotics Toolbox for Python, and the Swift Simulator to provide examples and implementations of algorithms. While not absolutely essential, for the most engaging and informative experience, we recommend working through the Jupyter Notebooks while reading this article. The Notebooks and setup instructions can be accessed at https://github.com/jhavl/dkt.

As the complexity of robot systems increases, it becomes more effective to simulate them before deployment. To do this, a model of the robot's kinematics or dynamics is required, and the most commonly used format is the Unified Robot Description Format (URDF). This article presents, to our knowledge, the first dataset of URDF files from various industrial and research organizations, with metadata describing each robot, its type, manufacturer, and the source of the model. The dataset contains 322 URDF files of which 195 are unique robot models, meaning the excess URDFs are either of a robot that is multiply defined across sources or URDF variants of the same robot. We analyze the files in the dataset, where we, among other things, provide information on how they were generated, which mesh file types are most commonly used, and compare models of multiply defined robots. The intention of this article is to build a foundation of knowledge on URDF and how it is used based on publicly available URDF files. Publishing the dataset, analysis, and the scripts and tools used enables others using, researching or developing URDFs to easily access this data and use it in their own work.

Vision is an effective sensor for robotics from which we can derive rich information about the environment: the geometry and semantics of the scene, as well as the age, identity, and activity of humans within that scene. This raises important questions about the reach, lifespan, and misuse of this information. This paper is a call to action to consider privacy in robotic vision. We propose a specific form of inherent privacy preservation in which no images are captured or could be reconstructed by an attacker, even with full remote access. We present a set of principles by which such systems could be designed, employing data-destroying operations and obfuscation in the optical and analogue domains. These cameras never see a full scene. Our localisation case study demonstrates in simulation four implementations that all fulfil this task. The design space of such systems is vast despite the constraints of optical-analogue processing. We hope to inspire future works that expand the range of applications open to sighted robotic systems.

Robotics is powered by software. Software tools control the rate of innovation in robotics research, drive the growth of the robotics industry, and power the education of future innovators and developers. Nearly 900,000 open-source repositories on GitHub are tagged with the keyword robotics—a potentially vast resource, but only a fraction of those are truly accessible in terms of quality, licensability, understandability, and total cost of ownership. The challenge is to match this resource to the needs of students, researchers, and companies to power cutting-edge research and real-world industrial solutions. This article reviews software tools for robotics, including both those created by the community at large and those created by the authors, as well as their impact on education, research, and industry. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 7 is May 2024. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Grasping compliant objects is difficult for robots - applying too little force may cause the grasp to fail, while too much force may lead to object damage. A robot needs to apply the right amount of force to quickly and confidently grasp the objects so that it can perform the required task. Although some methods have been proposed to tackle this issue, performance assessment is still a problem for directly measuring object property changes and possible damage. To fill the gap, a new concept is introduced in this paper to assess compliant robotic grasping using instrumented objects. A proof-of-concept design is proposed to measure the force applied on a cuboid object from a first-object perspective. The design can detect multiple contact locations and applied forces on its surface by using multiple embedded 3D Hall sensors to detect deformation relative to embedded magnets. The contact estimation is achieved by interpreting the Hall-effect signals using neural networks. In comprehensive experiments, the design achieved good performance in estimating contacts from each single face of the cuboid and decent performance in detecting contacts from multiple faces when being used to evaluate grasping from a parallel jaw gripper, demonstrating the effectiveness of the design and the feasibility of the concept.

This is the second and final article on the tutorial on manipulator differential kinematics. In Part 1, we described a method of modelling kinematics using the elementary transform sequence (ETS), before formulating forward kinematics and the manipulator Jacobian. We then described some basic applications of the manipulator Jacobian including resolved-rate motion control (RRMC), inverse kinematics (IK), and some manipulator performance measures. In this article, we formulate the second-order differential kinematics, leading to a definition of manipulator Hessian. We then describe the differential kinematics' analytical forms, which are essential to dynamics applications. Subsequently, we provide a general formula for higher-order derivatives. The first application we consider is advanced velocity control. In this section, we extend resolved-rate motion control to perform sub-tasks while still achieving the goal before redefining the algorithm as a quadratic program to enable greater flexibility and additional constraints. We then take another look at numerical inverse kinematics with an emphasis on adding constraints. Finally, we analyse how the manipulator Hessian can help to escape singularities. We have provided Jupyter Notebooks to accompany each section within this tutorial. The Notebooks are written in Python code and use the Robotics Toolbox for Python, and the Swift Simulator to provide examples and implementations of algorithms. While not absolutely essential, for the most engaging and informative experience, we recommend working through the Jupyter Notebooks while reading this article. The Notebooks and setup instructions can be accessed at https://github.com/jhavl/dkt.

Melt electrowriting (MEW) is an emerging high-resolution 3D printing technology used in biomedical engineering, regenerative medicine, and soft robotics. Its transition from academia to industry faces challenges such as slow experimentation, low printing throughput, poor reproducibility, and user-dependent operation, largely due to the nonlinear and multiparametric nature of the MEW process. To address these challenges, we applied computer vision and machine learning to monitor and analyze the process in real-time through imaging of the MEW jet between the nozzle-collector gap. To collect data for training we developed an automated data collection methodology that eases the experimental time from days to hours. A feedforward neural network, working in concert with optimization methods and a feedback loop, is used to develop closed-loop control ensuring reproducibility of the printed parts. We demonstrate that machine learning allows streamlining the MEW operation via closed-loop control of the highly nonlinear 3D printing technology.

Fabric manipulation is a long-standing challenge in robotics due to the enormous state space and complex dynamics. Learning approaches stand out as promising for this domain as they allow us to learn behaviours directly from data. Most prior methods however rely heavily on simulation, which is still limited by the large sim-to-real gap of deformable objects or rely on large datasets. A promising alternative is to learn fabric manipulation directly from watching humans perform the task. In this work, we explore how demonstrations for fabric manipulation tasks can be collected directly by humans, providing an extremely natural and fast data collection pipeline. Then, using only a handful of such demonstrations, we show how a pick-and-place policy can be learned and deployed on a real robot, without any robot data collection at all. We demonstrate our approach on a fabric smoothing and folding task, showing that our policy can reliably reach folded states from crumpled initial configurations. Code, video and data are available on the project website: https://sites.google.com/view/foldingbyhand

We present a reactive base control method that enables high performance mobile manipulation on-the-move in real-world environments with static and dynamic obstacles. Performing manipulation tasks while the mobile base remains in motion can significantly decrease the time required to perform multi-step tasks, as well as improve the gracefulness of the robot's motion. Existing approaches to manipulation on-the-move either ignore the obstacle avoidance problem or rely on the execution of planned trajectories, which is not suitable in environments with dynamic objects and obstacles. The presented controller addresses both of these deficiencies and demonstrates robust performance of pick-and-place tasks in dynamic environments. The performance is evaluated on several simulated and real-world tasks. On a mobile manipulation task with static obstacles, we outperform an existing method by 41% in terms of total task time. Further, we present real-world examples of our robot performing manipulation tasks on-the-move while avoiding a second autonomous robot in the workspace.

The task in visual servoing is to control the pose of the robot’s end-effector, relative to the goal, using visual features extracted from an image of the goal object. As shown in Fig. 15.1 the camera may be carried by the robot or be fixed in the world. The configuration of Fig. 15.1a has the camera mounted on the robot’s end-effector observing the goal, and is referred to as end-point closedloop or eye-in-hand. The configuration of Fig. 15.1b has the camera at a fixed point in the world observing both the goal and the robot’s end-effector, and is referred to as end-point open-loop. In the remainder of this book we will discuss only the eye-in-hand configuration.

With the increasing complexity of robot systems, it is necessary to simulate them before deployment. To do this, a model of the robot's kinematics or dynamics is required. One of the most commonly used formats for modeling robots is the Unified Robot Description Format (URDF). The goal of this article is to understand how URDF is currently used, what challenges people face when working with it, and how the community sees the future of URDF. The outcome can potentially be used to guide future research. This article presents the results from a survey based on 510 anonymous responses from robotic developers of different backgrounds and levels of experience. We find that 96.8% of the participants have simulated robots before, and of them 95.5% have used URDF. We identify a number of challenges that complicate the use of URDF, such as the inability to model parallel linkages and closed-chain systems, no real standard, lack of documentation, and a limited number of dynamic parameters to model the robot. Future perspectives for URDF are also determined, where 53.5% believe URDF will be more commonly used in the future, 12.2% believe other standards or tools will make URDF obsolete, and 34.4% are not sure what the future of URDF will be. Most participants agree there is a need for better tools to ensure URDF's future use.

Workpiece placement with respect to an industrial robot plays an important role in robotic manufacturing due to its influence on the configuration-dependent properties of industrial robots. Suboptimal placements of the workpiece may increase the required joint torques and decrease the dexterity of the robot. The focus of this work is to identify an optimal workpiece pose that enables a robot to carry out surface finishing with configurations that require the lowest possible joint torques while having maximum possible manipulability. We present a non-linear optimization-based algorithm to solve this problem and demonstrate the algorithm's capability on different workpieces which we share to facilitate further research in this area.

Finger-tip tactile sensors are increasingly used for robotic sensing to establish stable grasps and to infer object properties. Promising performance has been shown in a number of works for inferring adjectives that describe the object, but there remains a question about how each taxel contributes to the performance. This paper explores this question with empirical experiments, leading insights for future finger-tip tactile sensor usage and design: one tactile sensor instead of a pair of sensors is sufficient for symmetric objects and interaction motions; dense taxels are beneficial for texture-related adjectives, but can be distracting to non-texture-related ones; and a frame-rate much lower than the BioTac sensor can satisfy the demand of inferring object adjectives in the PHAC-2 dataset.

Robotic vision, otherwise known as computer vision for robots, is a critical process for robots to collect and interpret detailed information related to human actions, goals, and preferences, enabling robots to provide more useful services to people. This survey and systematic review presents a comprehensive analysis on robotic vision in human-robot interaction and collaboration (HRI/C) over the past 10 years. From a detailed search of 3,850 articles, systematic extraction and evaluation was used to identify and explore 310 papers in depth. These papers described robots with some level of autonomy using robotic vision for locomotion, manipulation, and/or visual communication to collaborate or interact with people. This article provides an in-depth analysis of current trends, common domains, methods and procedures, technical processes, datasets and models, experimental testing, sample populations, performance metrics, and future challenges. Robotic vision was often used in action and gesture recognition, robot movement in human spaces, object handover and collaborative actions, social communication, and learning from demonstration. Few high-impact and novel techniques from the computer vision field had been translated into HRI/C. Overall, notable advancements have been made on how to develop and deploy robots to assist people.