To explore safe interactions between a mobile robot and dynamic obstacles, this letter presents a comprehensive approach to collision-free navigation in dynamic indoor environments. The approach integrates Multimodal Motion Predictions (MMPs) of dynamic obstacles with predictive control for obstacle avoidance. MMP is achieved by a deep-learning method that predicts multiple plausible future positions. By repeating the MMP for each time offset in the future, multi-time-step MMPs are obtained. A nonlinear Model Predictive Control (MPC) solver uses the prediction outcomes to achieve collision-free trajectory tracking for the mobile robot. The proposed integration of multimodal motion prediction and trajectory tracking outperforms other non-deep-learning methods in complex scenarios. The approach enables safe interaction between the mobile robot and stochastic dynamic obstacles.

Learning long-horizon manipulation tasks such as stacking, presents a longstanding challenge in the field of robotic manipulation, particularly when using Reinforcement Learning (RL) methods. RL algorithms focus on learning a policy for executing the entire task instead of learning the correct sequence of actions required to achieve complex goals. While RL aims to find a sequence of actions that maximises the total reward of the task, the main challenge arises when there are infinite possibilities of chaining these actions (e.g. reach, grasp, etc.) to achieve the same task (stacking). In these cases, RL methods may struggle to find the optimal policy. This paper introduces a novel framework that integrates the operator concepts from the symbolic planning domain with hierarchical RL methods. We propose to change the way complex tasks are trained by learning independent policies of the actions defined by high-level operators instead of learning a policy for the complete complex task. Our contribution integrates planning operators (e.g. preconditions and effects) as part of the hierarchical RL algorithm based on the Scheduled Auxiliary Control (SAC-X) method. We developed a dual-purpose high-level operator, which can be used both in holistic planning and as independent, reusable policies. Our approach offers a flexible solution for long-horizon tasks, e.g., stacking and inserting a cube. The experimental results show that our proposed method achieved an average success rate of 97.2% for learning and executing the whole stack. Furthermore, we obtain a high success rate when learning independent policies, e.g. reach (98.9%), lift (99.7%), move (97.4%), etc. The training time is also reduced by 68% when using our proposed approach.

Traditional robot navigation passively plans/replans to avoid any contact with obstacles in the scene. This limits the obtained solutions to the collision‐free space and leads to failures if the path to the goal is obstructed. In contrast, humans actively modify their environment by repositioning objects if it assists locomotion. This article aims to bring robots closer to such abilities by providing a framework to detect and clear movable obstacles to continue navigation. The approach leverages a multimodal robot skin that provides both local proximity and tactile feedback regarding physical interactions with the surroundings. This multimodal contact feedback is employed to adapt the robot's behavior when interacting with object surfaces and regulating applied forces. This enables the robot to remove bulky obstacles from its path and solves otherwise infeasible navigation problems. The system's ability is demonstrated in simulation and real‐world scenarios involving movable and nonmovable obstacles.

Human-robot interaction (HRI) describes scenarios in which both human and robot work as partners, sharing the same environment or complementing each other on a joint task. HRI is characterized by the need for high adaptability and flexibility of robotic systems toward their human interaction partners. One of the major challenges in HRI is task planning with dynamic subtask assignment, which is particularly challenging when subtask choices of the human are not readily accessible by the robot. In the present work, we explore the feasibility of using electroencephalogram (EEG) based neuro-cognitive measures for online robot learning of dynamic subtask assignment. To this end, we demonstrate in an experimental human subject study, featuring a joint HRI task with a UR10 robotic manipulator, the presence of EEG measures indicative of a human partner anticipating a takeover situation from human to robot or vice-versa. The present work further proposes a reinforcement learning based algorithm employing these measures as a neuronal feedback signal from the human to the robot for dynamic learning of subtask-assignment. The efficacy of this algorithm is validated in a simulation-based study. The simulation results reveal that even with relatively low decoding accuracies, successful robot learning of subtask-assignment is feasible, with around 80% choice accuracy among four subtasks within 17 minutes of collaboration. The simulation results further reveal that scalability to more subtasks is feasible and mainly accompanied with longer robot learning times. These findings demonstrate the usability of EEG-based neuro-cognitive measures to mediate the complex and largely unsolved problem of human-robot collaborative task planning.

The next generation of robots is expected to work collaboratively with humans in natural (dynamic) settings. For this, it is important to properly study and model human factors, so that the AI and Robotic models can include them to enable robust Human-Robot Collaborations. This will enable safe and trustworthy hybrid decision-making approaches – Responsible AI – thereby streamlining robust collaborations (as per human-centred expectations). This interdisciplinary workshop will focus on the intersection of Cognitive Human Factors, Interpretable & Explainable AI methods, Social Interaction, and Human-Centred Robotics to stimulate novel long-range avenues for innovative human-centred collaborative methods in real-world contexts.

Understanding the human brain's perception of different thermal sensations has sparked the interest of many neuroscientists. The identification of distinct brain patterns when processing thermal stimuli has several clinical applications, such as phantom-limb pain prediction, as well as increasing the sense of embodiment when interacting with neurorehabilitation devices. Notwithstanding the remarkable number of studies that have touched upon this research topic, understanding how the human brain processes different thermal stimuli has remained elusive. More importantly, very intense thermal stimuli perception dynamics, their related cortical activations, as well as their decoding using effective features are still not fully understood. In this study, using electroencephalography (EEG) recorded from three healthy human subjects, we identified spatial, temporal, and spectral patterns of brain responses to different thermal stimulations ranging from extremely cold and hot stimuli (very intense), moderately cold and hot stimuli (intense), to a warm stimulus (innocuous). Our results show that very intense thermal stimuli elicit a decrease in alpha power compared to intense and innocuous stimulations. Spatio-temporal analysis reveals that in the first 400 ms post-stimulus, brain activity increases in the prefrontal and central brain areas for very intense stimulations, whereas for intense stimulation, high activity of the parietal area was observed post-500 ms. Based on these identified EEG patterns, we successfully classified the different thermal stimulations with an average test accuracy of 84% across all subjects. En route to understanding the underlying cortical activity, we source localized the EEG signal for each of the five thermal stimuli conditions. Our findings reveal that very intense stimuli were anticipated and induced early activation (before 400 ms) of the anterior cingulate cortex (ACC). Moreover, activation of the pre-frontal cortex, somatosensory, central, and parietal areas, was observed in the first 400 ms post-stimulation for very intense conditions and starting 500 ms post-stimuli for intense conditions. Overall, despite the small sample size, this work presents novel findings and a first comprehensive approach to explore, analyze, and classify EEG-brain activity changes evoked by five different thermal stimuli, which could lead to a better understanding of thermal stimuli processing in the brain and could, therefore, pave the way for developing a real-time withdrawal reaction system when interacting with prosthetic limbs. We underpin this last point by benchmarking our EEG results with a demonstration of a real-time withdrawal reaction of a robotic prosthesis using a human-like artificial skin.

For physical human–robot interaction (pHRI) where multi‐contacts play a key role, both robustness to achieve robot‐intended motion and adaptability to follow human‐intended motion are fundamental. However, there are tradeoffs during pHRI when their intentions do not match. This paper focuses on bipedal walking control during pHRI, which handles such tradeoff when a human and a humanoid robot having different footsteps locations and durations. To resolve this, a force‐reactive walking controller is proposed by adequately combining ankle and stepping strategies. The ankle strategy maintains the robot's intention based on an analytically‐optimal center of pressure, leading the robot to oppose resistance to multiple contacts from the human. Based on the robot's kinodynamic constraints and/or the confidence of the robot's intention, the stepping strategy updates the robot's footsteps based on the human's intention implied by the multiple contact forces. Consequently, the proposed walking control on pHRI mutually exchanges human–robot intentions in real‐time, thereby achieving coordinated steps. With a full‐sized humanoid robot that is able to detect multi‐contacts in real‐time, we succeeded in performing a long‐term “box‐step” with multi‐contacts pHRI, demonstrating the robustness of our approach.

Providing sensitive skin to robots has been explored since the 1980s [1]. The reasons for artificial robot skin are diverse. Recently, new development and new application of robot skin have seen another boost, because collaborative and interactive robots have been considered as a viable solution (i) to further increase the level of automation in complex industrial scenarios [2]; (ii) for health-care [3]; and (iii) also in household [4] applications. So, why do robots need sensitive skin? Skin is considered to be the key factor to (1) enable robots to recognize textures for contact/object classification and recognition and (2) enable intuitive and safe human-robot interaction and collaboration. Texture recognition allows robots to add feel to objects, which so far are only known visually. For example, the visual knowledge of an object (round and yellow) can be extended with the feel of the object (soft with a smooth surface). The knowledge helps the robot to increase the success rate of interaction and manipulation tasks because the robot can exploit knowledge about the grip properties of the object through a sense of touch. Furthermore, artificial robot skin can guarantee the safety of humans in the robots' workspace. In contrast to visual safeguards, contacts are direct and cannot be occluded. In this way, robot skin already contributes to collaborative robots. Potentially, it makes robots safe enough to remove safety fences and allow humans to touch and interact closely with the robot. In addition to that, robot skin can provide an intuitive interface for manipulating and teaching the robot. With the development of appropriate tactile behaviors, the robot can be guided and taught simply by touching and moving it as desired [5].

This work proposes and realizes a control architecture that can support the deployment of a large‐scale robot skin in a Human‐Robot Collaboration scenario. It is shown, how whole‐body tactile feedback can extend the capabilities of robots during dynamic interactions by providing information about multiple contacts across the robot's surface. Specifically, an uncalibrated skin system is used to implement stable force control while simultaneously handling the multi‐contact interactions of a user. The system formulates control tasks for force control, tactile guidance, collision avoidance, and compliance, and fuses them with a multi‐priority redundancy resolution strategy. The approach is evaluated on an omnidirectional mobile‐manipulator with dual arms covered with robot skin. Results are assessed under dynamic conditions, showing that multi‐modal tactile information enables robust force control while at the same time remaining responsive to a user's interactions.

Making accurate predictions about the dynamic environment is crucial for the trajectory planning of mobile robots. Predictions are by nature uncertain, and for motion prediction multiple futures are possible for the same historic behavior. In this work, the objective is to predict possible future positions of the target object for the collision avoidance purpose for mobile robots by considering different uncertainty by combining a sampling-based idea with data-driven methods. More specifically, we propose a major improvement on a loss function for multiple hypotheses and test it with convolutional neural networks on motion prediction problems. We implement post-processing heuristics that produce multiple Gaussian distribution estimations, and show that the result is suitable for trajectory planning for mobile robots. The method is also evaluated with the Stanford Drone Dataset.

Physical therapy is the pillar of rehabilitation for disabilities caused by neurological disorders, however not every patient has access to it due to the lack of human resources. Robot-based rehabilitation can contribute to the solution of this challenge with benefits such as task-oriented exercise routines. In order to increase therapy frequency and intensity, we propose the use of robot skin as multi-sensory interface to maximize the wearability and support of a lightweight elbow flexion/extension exoskeleton. The robot skin exoskeleton detects the motion intention by measuring acceleration, proximity, and interaction forces. This allows the implementation of control modes which are inspired by physical therapy such as passive movement, active support, resistance training, and corrective therapy. To assess every therapy-inspired control mode, a study analyzing force readings and sEMG recordings of the biceps during exoskeleton use was conducted with four healthy subjects to test the exoskeleton functionality. Our results show that higher assistive levels under the supportive therapy control modes result in larger reductions of normalized sEMG ( $>$ 40% in passive exercises), whereas higher resistive levels result in an increase of normalized sEMG ( $>$ 30% in resistive exercises). Thus, enabling the exoskeleton to use multi-sensory interfaces to implement therapy routines based on user intention.

Multi-agent robot systems, specifically mobile robots in dynamic environments interacting with humans, e.g., assisting in production environments, have seen an increased interest over the past years. To better understand the ROS2 communication in a network with a high load of nodes, this paper investigates the communication handling of multiple robots to a single tracking node for centralized multi-agent robot systems using ROS2. Thereore, a quantitative analysis of two publisher-subscriber communication architectures and a comparative study between DDS vendors (CycloneDDS, FastDDS and GurumDDS) using ROS2 Galactic is performed. The architectures of consideration are a many-to-one approach, where multiple robots communicate to a central node over one topic, and the one-to-one communication approach, where multiple robots communicate over particular topics to a central node. Throughout this work, the increase in the number of robots at different publishing rates is simulated on a single computer for the different DDS vendors. A further simulation is done using a distributed setup with CycloneDDS. The simulations show that with an increase in the number of nodes, the average data age and the data miss ratio in the one-to-one approach were significantly lower than in the many-to-one approach. CycloneDDS was shown as the most robust regarding crashes and response time under system launch, while FastDDS showed better results regarding the data ageing.

The ability to decide and adjust actions according to motion prediction of dynamic obstacles offers a flexible planning scheme and ampler reaction time to avoid potential impact. Prediction-based collision avoidance implies a two-stage decision-making process from motion prediction to action planning. One of the challenges in motion prediction is the movements of objects are usually non-deterministic and governed by multimodal models. Many studies have been made on motion prediction of dynamic obstacles and action planning for mobile robots separately. The objective of this work is to explore their coherence in terms of multiple future predictions by combining a data-driven motion prediction approach with a model-based control strategy. More specifically, we integrate motion prediction from deep learning models, Mixture Density Networks (MDNs) with a Non-linear Model Predictive Control (NMPC) framework. The deep learning models produce the multimodal probability distribution of future positions of dynamic obstacles, which is utilized by the MPC controller as a constraint. We show via simulation that the selected model provides valid predictions of motion in a dynamic environment. The prediction result endows the controller with the capability to avoid dynamic obstacles in advance.

Physical therapy is the pillar of rehabilitation for disabilities caused by neurological disorders, however not every patient has access to it due to the lack of human resources. Robot-based rehabilitation devices such as exoskeletons can contribute to the solution of this challenge by providing benefits such as task-oriented exercise routines. In order to increase the therapy frequency and intensity, we propose the use of robot skin as multi-sensory interface to maximize the wearability and support of a lightweight upper-limb exoskeleton for elbow flexion and extension. The robot skin covered exoskeleton measures acceleration, proximity, and interaction forces, allowing the implementation of different control modes which are inspired by physical therapy such as passive movement, active support, and resistance training. To assess every therapy-inspired control mode, a preliminary study analyzing sEMG recordings of the biceps during exoskeleton use was conducted with four healthy subjects to test the functionality of the exoskeleton. Our results show that higher assistive levels under the supportive therapy control modes result in larger reductions of normalized sEMG (>40% in passive exercises), whereas higher resistive levels result in an increase of normalized sEMG (>30% in resistive exercises). Thus, enabling the exoskeleton to use multi-sensory interfaces to implement therapy support routines.

In a physical Human-Robot Interaction for industrial scenarios is paramount to guarantee the safety of the user while keeping the robot's performance. Hierarchical task approaches are not sufficient since they tend to sacrifice the low priority tasks in order to guarantee the consistency of the main task. To handle this problem, we enhance the standard hierarchical fusion by introducing a novel interactive task-reconfiguring approach (TACTO-Selector) that uses the information of the tactile interaction to adapt the dimension of the tasks, therefore guaranteeing the execution of the safety task while performing the other task as good as possible. In this work, we hierarchically combine a 6 DOF Position-Based Visual Servoing (PBVS) task with a reactive skin control. This approach was evaluated on a 6 DOF industrial robot showing an improvement of 36.37% on average in tracking error reduction compared with a standard approach.

This work introduces a new sensing system for biped robots based on plantar robot skin, which provides not only the resultant forces applied on the ankles but a precise shape of the pressure distribution in the sole together with other extra sensing modalities (temperature, pre-touch and acceleration). The information provided by the plantar robot skin can be used to compute the center of pressure and the ground reaction forces. This information also enables the online construction of the supporting polygon and its preemptive shape before foot landing using the proximity sensors in the robot skin. Two experiments were designed to show the advantages of this new sensing technology for improving balance and walking controllers for biped robots over unknown terrain.

The sense of touch enables us to safely interact and control our contacts with our surroundings. Many technical systems and applications could profit from a similar type of sense. Yet, despite the emergence of e-skin systems covering more extensive areas, large-area realizations of e-skin effectively boosting applications are still rare. Recent advancements have improved the deployability and robustness of e-skin systems laying the basis for their scalability. However, the upscaling of e-skin systems introduces yet another challenge—the challenge of handling a large amount of heterogeneous tactile information with complex spatial relations between sensing points. We targeted this challenge and proposed an event-driven approach for large-area skin systems. While our previous works focused on the implementation and the experimental validation of the approach, this work now provides the consolidated foundations for realizing, designing, and understanding large-area event-driven e-skin systems for effective applications. This work homogenizes the different perspectives on event-driven systems and assesses the applicability of existing event-driven implementations in large-area skin systems. Additionally, we provide novel guidelines for tuning the novelty-threshold of event generators. Overall, this work develops a systematic approach towards realizing a flexible event-driven information handling system on standard computer systems for large-scale e-skin with detailed descriptions on the effective design of event generators and decoders. All designs and guidelines are validated by outlining their impacts on our implementations, and by consolidating various experimental results. The resulting system design for e-skin systems is scalable, efficient, flexible, and capable of handling large amounts of information without customized hardware. The system provides the feasibility of complex large-area tactile applications, for instance in robotics.

This letter evaluates and describes the large-scale integration of our multi-modal event-driven robot skin system on our humanoid robot H1 (REEM-C, PAL robotics). The robot skin is powered by the robot and all processing of tactile perception and control are executed onboard the robot. The robot skin system largely covers the humanoid and employs 1260 skin cells in 47 skin patches, in total 7560 multi-modal tactile sensors. The robot skin system is driven by events, i.e., the occurrence of novel information drives the acquisition, transmission, and processing of information rather than the synchronous sampling clock as in clock-driven systems. The event-driven robot skin system enables efficient multi-modal large area tactile perception, which is a prerequisite for realizing reactive whole-body control. We analyze the new robot skin system and evaluate its performance in clock-driven mode and event-driven mode. This analysis includes modeling the required CPU load and investigating the scaling of the system in multi-core systems. We evaluate the robot skin system with an experiment, where a large number of skin cells is activated (>680 cells), and a large number of events is generated. The obtained results demonstrate the superior performance of the event-driven system. In the clock-driven mode, the robot skin system constantly produces 315 000 packets/s, while in event-driven mode the system at most produces 40 000 packets/s (13%). The CPU load reduces from constantly 270% to at most 100% (37%). In clock-driven mode, the PC drops on average 80 000 packets/s (25% of all packets), while in event-driven mode, the package loss is practically neglectable. This efficient large-scale robot skin enables the complete onboard integration into a humanoid robot without the need for additional external power or processing capabilities.

This article presents a holistic approach to the engineering of an artificial robot skin for robots. An example of a multimodal skin cell is given, one that supports multiple human-like sensing modalities, and support for skin cell network is also provided; this is essential to form large-area skin patches in order to cover the surfaces of robots. The essential elements of efficiently handling a large amount of tactile data are explained. A general control framework, which supports robots commanded in position, velocity, and torque, is provided and validated. Several applications of this robot skin will be presented, demonstrating the effectiveness and efficiency of our artificial robot skin to support a wide number of robotic platforms as well as its ease of use across different domains.

In this letter, we introduce our approach to walking assistance for elderly adults through predictive optimization of gait assistive force. We focus on providing supportive interaction force to the user during walking with a robotic assistive device with an admittance controlled mobile base. Appropriate physical human-robot interaction (pHRI) could be beneficial in reducing the risk associated with immobility such as disuse syndrome by encouraging physical activities with proper assistance. We propose an optimization algorithm based on a model predictive control approach in order to provide desirable assistive forces according to the estimated user's state during walking. Using a simplified human gait model with a linear inverted pendulum, we formulate the optimization of the assistive forces as a linear quadratic programming problem that can be suitable for real-time pHRI. Numerical simulations and experimental results demonstrate the feasibility of our gait support strategy in achieving appropriate compliant interactions during walking, fall prevention, and suitable positioning for user companion.