This research addresses shared intention prediction in multi-human, multi-robot environments. We propose an intention prediction pipeline based on Bayesian inference, enabling robots to predict humans’ navigational intent. Our pipeline uses prior semantic knowledge about potential goal destinations of the humans. We furthermore investigated different strategies for robots to share information to improve their predictions. A dataset was collected specifically for testing our pipeline and comparing different sharing strategies. The dataset consists of camera feeds from two robots observing two humans performing simple pick-up tasks. The pipeline correctly predicts 63

We present a framework for the automatic configuration of constraints in a disturbed model predictive control (MPC) formulation for multi-agent trajectory generation, leveraging semantic knowledge encoded in a robot world model. Agent routes are planned as sequences of semantic areas to be traversed, enabling the systematic selection of relevant environmental and inter-agent constraints through queries of the world model. The introduction of sub-areas, such as lanes, further reduces computational complexity by limiting the activation of coupling constraints. In addition, the framework supports reasoning over discrete actions, such as yielding to other agents, allowing agents to satisfy high-level task and environmental constraints within the continuous MPC formulation. The proposed framework is validated in both simulation and hardware-in-the-loop experiments using a distributed computing setup. The results demonstrate improved task execution performance while significantly reducing computation time when compared with baseline centralized and distributed formulations.

Teleoperation for contact-rich manipulation remains challenging, especially when using low-cost, motion-only interfaces that provide no haptic feedback. Virtual reality controllers enable intuitive motion control but do not allow operators to directly perceive or regulate contact forces, limiting task performance. To address this, we propose an augmented reality (AR) visualization of the impedance controller's target pose and its displacement from each robot end effector. This visualization conveys the forces generated by the controller, providing operators with intuitive, real-time feedback without expensive haptic hardware. We evaluate the design in a dual-arm manipulation study with 17 participants who repeatedly reposition a box with and without the AR visualization. Results show that AR visualization reduces completion time by 24

We present a framework for the automatic configuration of constraints in a disturbed model predictive control (MPC) formulation for multi-agent trajectory generation, leveraging structural knowledge encoded in a robot world model. Agent routes are planned as sequences of areas to be traversed, enabling the systematic selection of relevant environmental and inter-agent constraints through queries of the world model. The introduction of subareas, such as lanes, further reduces computational complexity by limiting the activation of coupling constraints. In addition, the framework supports reasoning over discrete actions, such as yielding to other agents, allowing agents to satisfy high-level task and environmental constraints within the continuous MPC formulation. The proposed framework is validated in both simulation and hardware-in-the-loop experiments using a distributed computing setup. The results demonstrate improved task execution performance while significantly reducing computation time when compared with baseline distributed formulations.

This paper studies the zero dynamics stability of a multirotor Unmanned Aerial Vehicle (UAV) carrying a rigidly attached heavy payload. In particular, we focus on controlling the position of a virtual task-relevant point of interest (POI) on the UAV and investigate how its stability depends on payload placement. Taking the POI position and the yaw angle as output, we prove that the zero dynamics are Lyapunov stable only if this POI is above the system's total center of gravity, assuming that the POI and the payload are both on the UAV's body-fixed z-axis. Remarkably, this indicates that when the payload coincides with the POI, zero dynamics stability is only guaranteed if the payload is above the UAV rather than below it. For the more general case where the POI and payload do not necessarily lie on the same z-axis, we demonstrate stability for a simplified planar model. The analysis also yields an explicit bound on the horizontal payload offset given the rotor thrust limits. Finally, numerical MATLAB simulations provide insight into how results for the planar model are also observed for the general 3D system.

Humans typically convey route instructions as a sequence of locations where actions need to be executed, e.g., turn left at the second crossing. It is not uncommon that humans generalize or forget parts of the environment or route instructions. Consequently, route instructions can be incomplete as decision points, in the form of action-location descriptions, are omitted. In addition, omitted location descriptions may be similar to one already present in the route instructions, thus making it ambiguous to the wayfinder at which specific location to execute the action. Defensive wayfinding then characterizes the procedure to deal with such uncertainties in the route instructions. The state of art is not capable of performing defensive wayfinding for mobile robots with route instructions that are incomplete and ambiguous regarding the specific location for action execution. This work tackles this problem by taking inspiration from practices of human wayfinding literature and incorporating it into the robotics context. More in particular by adding three types of knowledge to the route instructions: (1) the types of locations that can be encountered by the robot, (2) the action models to leave those locations, and (3) the temporal and spatial constraints on the (topological) sequence of encountered locations. (In the context of human defensive wayfinding, this additional information is “background knowledge”, or it is provided in the form of a rough sketch.) Our defensive wayfinding approach relies on a hypothesis tree that associates (parts of) the executed robot path to the route instructed path. We experimentally validate defensive wayfinding in simulation with a mobile robot equipped with a 2D laser range finder in a corridor-junction environment that can be topologically inconsistent with the provided route instructions.

This paper presents an innovative approach to semantic path planning for mobile robots by integrating semantic data from building digital twins. Semantic and metric information extracted from the digital twin is used to assign weights to a connectivity graph, allowing for path computation using the A* algorithm. Our method excels at generating robot-specific maps that combine both geometric and semantic data, diverging from traditional static maps. This semantic integration equips robots with diverse navigation skills, enabling them to navigate complex environments within large smart facilities. A key innovation of this work is our path planner, which utilizes semantic data from Building Information Modeling (BIM) databases. This marks a significant advancement in mobile robotic navigation, accommodating robots with varying navigation abilities. The significance of this work lies in the seamless integration of semantic data, enhancing the adaptability and efficiency of mobile robots, regardless of their navigation skills. This coordinated navigation system not only improves safety but also optimizes shared space management for both humans and robots.

This paper introduces a novel approach that integrates future closest point predictions into the distance constraints of a collision avoidance controller, leveraging convex hulls with closest point distance calculations. By addressing abrupt shifts in closest points, this method effectively reduces collision risks and enhances controller performance. Applied to an Image Guided Therapy robot and validated through simulations and user experiments, the framework demonstrates improved distance prediction accuracy, smoother trajectories, and safer navigation near obstacles.

Robotization is considered a key solution to labor shortages in the agri-food industry. However, deploying robots in natural environments is challenging due to unpredictable factors such as plant variances and occlusions. This paper focuses on the localization of tomato trusses for autonomous harvesting by servoing a robot-mounted camera to different viewpoints. We build on previous work where the robot is provided with prior knowledge of the tomato plant. Specifically, the geometric relations between the trusses are modeled as ranges, which reflect uncertainty. Our main contribution is an approach that represents this uncertainty as polytope volumes. Polytopes enable scalable reasoning that facilitates likelihood estimation for viewpoint selection. Our method first constructs polytope hypotheses regarding the truss locations based on prior plant knowledge. It then refines the polytope shapes using Bayesian updates based on camera observations. Finally, the polytopes are used to select the next viewpoint that maximizes the chance of observing a new tomato truss. Experiments show that polytope-based viewpoint selection speeds up truss localization compared to earlier methods, advancing robotic harvesting.

RoboCup soccer robots seldom use offline defined strategies to proactively coordinate their passing and movement actions in a soccer game. Human team sports do that abundantly, especially in Invasion Team Sports (ITS). This manuscript uses literature on ITS strategy, with a focus on futsal because of its similarity to RoboCup soccer, to analyze the tactical principles – imagined as abstract guidelines, e.g., “to threaten defensive lines” – behind their coordination patterns. Our main contribution is in the structured design of robot soccer strategies via an explicit composition of human-inspired tactical principles. Our approach provides robot soccer strategists with clearer insight in applying best practices of human strategy design, serving as a practical starting point for designing and adjusting robot soccer strategies; this is currently missing in robot soccer literature, as it does not delve deeper into why the coordination patterns exist. Our contribution is realized by performing the following steps: 1) the proposition of an initial guideline to apply a composition of tactical principles in the design of robot soccer strategies, 2) the formalization of tactical principles into robot action and perception constraints, and 3) the design of a 5-on-5 robot soccer strategy with the insights from the previous two steps. The strategy is implemented in the Skill, Tactics, and Plays (STP) software as used by the Tech United Eindhoven soccer robots. The experimental results validate the applicability of our designed strategy. The first validation, in the form of a sensitivity analysis, is performed on the Tech United Eindhoven simulator, and the second is realized in a real game against Robotsports.

Robotic X-ray imaging systems enable autonomous inspection of the internal integrity of critical infrastructure. However, these systems often suffer from vibrations and unwanted movements that cause motion blur in the resulting radiographs. The impact of this motion blur is often unknown until the first prototype is available and even then requires extensive experimental testing to assess. In addition, tests involving radiation are time-consuming, demand specialized equipment, and pose inherent safety risks. In this work, we propose using X-ray simulation as a tool to complement and replace real images during the development of robotic inspection systems. Our method extends an existing X-ray simulation framework (gVirtualXray) to generate motion-blurred images from any type of motion, which are then validated against experimental data. The approach is applicable to various robotic systems and we demonstrate its use for a decoupled two-drone inspection system for wind turbine blades. This is one of the most demanding applications due to the high degree of freedom of the system components and relatively long exposure times. The simulator provides insights into the motion blur sensitivity of the design, helping among others, to pinpoint the most significant degrees of freedom that affect image quality. Finally, we highlight the potential of the simulator for early estimation of performance limits, generation of training datasets for machine learning algorithms, and optimization of system design without the need for physical prototypes. Both the datasets and the software implementation are publicly available.

This paper presents a deep reinforcement learning-based motion planning approach that uses the semantics (e.g., material or element type) of building elements to avoid areas near unsafe elements (e.g., doors that can open), referred to as risky zones. To this end, a costmap containing the risky zones is automatically generated by querying the semantics and geometry of these elements from the digital twin of a building. The robot is discouraged from entering these risky zones by defining a penalty within the reward function of a Soft Actor-Critic (SAC) deep reinforcement learning agent. The approach is validated in simulation by navigating three different environments. A comparison is made between a SAC agent trained with the costmap and one that is not. Results show that the agent using the costmap successfully avoids the risky zones in front of the doors. This research is a step towards automatically interpreting semantic elements stored in building digital twins for robot navigation.

To ensure safe and smooth human-robot interaction, autonomous robots operating around people often necessitate 3D human body orientation and bounding box estimation. An orientation of the human is generally defined by the motion around the yaw-axis. However, human-robot interaction tasks may necessitate different orientation definitions depending on the application, for instance an orientation can be defined using shoulder and hip joints. A 3D skeleton map, consisting of multiple 3D body joint coordinates, enables defining an orientation in diverse ways. However, estimating 3D skeleton maps from monocular images is computationally expensive. Existing approaches using large neural network models are impractical for real-time robot operation due to the limited onboard computation and power resources. In this paper, we automatically label the 3D human body orientation from 3D skeleton maps, and present a deep learning method to estimate 3D orientation and bounding box. We achieve this by leveraging a larger neural perception model to automatically generate an annotated training dataset, using a functional mapping from 3D skeleton joint coordinates to the defined orientation and bounding boxes. Experimental results demonstrate that our perception model estimates 3D human body orientation with an average error of 10.94 ^∘ around the yaw-axis and the 3D bounding box with errors of 19.15

This paper presents a framework for multi-agent navigation in structured but dynamic environments, integrating three key components: a shared semantic map encoding metric and semantic environmental knowledge, a claim policy for coordinating access to areas within the environment, and a Model Predictive Controller for generating motion trajectories that respect environmental and coordination constraints. The main advantages of this approach include: (i) enforcing area occupancy constraints derived from specific task requirements; (ii) enhancing computational scalability by eliminating the need for collision avoidance constraints between robotic agents; and (iii) the ability to anticipate and avoid deadlocks between agents. The paper includes both simulations and physical experiments demonstrating the framework's effectiveness in various representative scenarios.

The transition to robotic construction requires advanced planning approaches to meet the specific requirements for successful robot deployment. Robots require precise information (e.g., position, velocity, force) to execute tasks reliably. Existing research focuses on robot-oriented construction processes but lacks real-time planning integration with building and construction information. To overcome these challenges, this paper proposes the Robot Construction Action Node (RCAN), serving as a middle layer system between BIM-based smart components and the Robot Operating System (ROS). RCAN breaks down construction planning into ordered activities for flexible adaptation and direct robot control. By assigning these construction activities to components, the macro construction process is linked to the micro construction methods. This procedure is tested and evaluated using a Linked Building Data (LBD) data modeling approach. The proposed solution is validated through three robot construction scenarios in an earlier built simulation environment.

With the increase in the availability of Building Information Models (BIM) and (semi-) automatic tools to generate BIM from point clouds, we propose a world model architecture and algorithms to allow the use of the semantic and geometric knowledge encoded within these models to generate maps for robot localization and navigation. When heterogeneous robots are deployed within an environment, maps obtained from classical SLAM approaches might not be shared between all agents within a team of robots, e.g. due to a mismatch in sensor type, or a difference in physical robot dimensions. Our approach extracts the 3D geometry and semantic description of building elements (e.g. material, element type, color) from BIM, and represents this knowledge in a graph. Based on queries on the graph and knowledge of the skills of the robot, we can generate skill-specific maps that can be used during the execution of localization or navigation tasks. The approach is validated with data from complex build environments and integrated into existing navigation frameworks.

We propose a shared semantic map architecture to construct and configure Model Predictive Controllers (MPC) dynamically, that solve navigation problems for multiple robotic agents sharing parts of the same environment. The navigation task is represented as a sequence of semantically labeled areas in the map, that must be traversed sequentially, i.e. a route. Each semantic label represents one or more constraints on the robots' motion behaviour in that area. The advantages of this approach are: (i) an MPC-based motion controller in each individual robot can be (re-)configured, at runtime, with the locally and temporally relevant parameters; (ii) the application can influence, also at runtime, the navigation behaviour of the robots, just by adapting the semantic labels; and (iii) the robots can reason about their need for coordination, through analyzing over which horizon in time and space their routes overlap. The paper provides simulations of various representative situations, showing that the approach of runtime configuration of the MPC drastically decreases computation time, while retaining task execution performance similar to an approach in which each robot always includes all other robots in its MPC computations.

This paper presents a Bayesian optimization framework for the automatic tuning of shared controllers which are defined as a Model Predictive Control (MPC) problem. The proposed framework includes the design of performance metrics as well as the representation of user inputs for simulation-based optimization. The framework is applied to the optimization of a shared controller for an Image Guided Therapy robot. VR-based user experiments confirm the increase in performance of the automatically tuned MPC shared controller with respect to a hand-tuned baseline version as well as its generalization ability.

Learning from demonstration is an approach that allows users to personalize a robot’s tasks. While demonstrations often focus on conveying the robot’s motion or task plans, they can also communicate user intentions through object attributes in manipulation tasks. For instance, users might want to teach a robot to sort fruits and vegetables into separate boxes or to place cups next to plates of matching colors. This paper introduces a novel method that enables robots to learn the semantics of user demonstrations, with a particular emphasis on the relationships between object attributes. In our approach, users demonstrate essential task steps by manually guiding the robot through the necessary sequence of poses. We reduce the amount of data by utilizing only robot poses instead of trajectories, allowing us to focus on the task’s goals, specifically the objects related to these goals. At each step, known as a keyframe, we record the end-effector pose, object poses, and object attributes. However, the number of keyframes saved in each demonstration can vary due to the user’s decisions. This variability in each demonstration can lead to inconsistencies in the significance of keyframes, complicating keyframe alignment to generalize the robot’s motion and the user’s intention. Our method addresses this issue by focusing on teaching the higher-level goals of the task using only the required keyframes and relevant objects. It aims to teach the rationale behind object selection for a task and generalize this reasoning to environments with previously unseen objects. We validate our proposed method by conducting three manipulation tasks aiming at different object attribute constraints. In the reproduction phase, we demonstrate that even when the robot encounters previously unseen objects, it can generalize the user’s intention and execute the task.

Performing tasks in the agri-food industry can be complex due to the organic, unstructured environments and natural variation in product. Successfully implementing robotic systems in such contexts is highly dependent on their capacity to adapt to variability, unpredictability and incomplete sensor data. Given the multitude of potential unexpected scenarios, it is no longer satisfactory to create fixed plans and controllers for the robot to address a task. Instead, it is essential to model our knowledge of the task and of the robot's operating environment, which is the central concept referred to as a ‘world model’ (WM). In this chapter, we elaborate on the notion of a WM and demonstrate how various WM types can be applied to address tasks in the agri-food domain. In addition to the prevalent geometric world models, we provide an example of the application of a topological WM in an environment devoid of geometric structure. We discuss several topics to improve robustness of task execution, such as multi-modal redundant skills and the use of invariants to describe the world, rather than relying on exact parameters. We argue that robots require an explicit WM as a central component in their software architecture. The software implementation of the WM holds the necessary information about the environment and configuration parameters of the system which are necessary to configure action and detection skills and interpret sensor information. In the future, we argue that these WMs will enable robots to reason online about the situation at hand and on the correct actions to perform to progress the task. The explicit nature of the WM allows robots to share their knowledge, which is crucial when collaborating with humans or other robots.