The ability to track the 6D pose distribution of an object when a mobile manipulator robot is still approaching the object can enable the robot to pre-plan grasps that combine base and arm motion. However, tracking a 6D object pose distribution from a distance can be challenging due to the limited view of the robot camera. In this work, we present a framework that fuses observations from external stationary cameras with a moving robot camera and sequentially tracks it in time to enable 6D object pose distribution tracking from a distance. We model the object pose posterior as a multi-modal distribution which results in a better performance against uncertainties introduced by large camera-object distance, occlusions and object geometry. We evaluate the proposed framework on a simulated multi-view dataset using objects from the YCB data set. Results show that our framework enables accurate tracking even when the robot camera has poor visibility of the object.

Flexible manufacturing requires rapid deployment of solutions and minimal setup time to remain competitive. An essential attribute is the ability to control error levels, as failures can range from minor performance degradation to severe equipment damage. However, conventional deployment often involves extensive setup, data collection, model training or parameter tuning, and system testing, resulting in significant delays that hinder commercial feasibility. We propose a data engine which gathers data and improve its performance while executing the task. The data engine consists of two classifiers, a fast model prediction and expensive verification. First, a model prediction is performed and based on the confidence level of the prediction, the expensive verification can be used. By adjusting the confidence level, users can control the level of tolerable error. Our method is implemented on a real-world robotic insertion task, which uses force data for the model prediction. The system applies UMAP dimensionality reduction and uses Wilson-Score to compute the confidence bounds of the prediction. Results demonstrate the ability to learn and reduce the need for expensive verifications over time, while staying within the set error-rate. The results highlight the potential of confidence bounds in self-improving models to enhance reliability in robotic classification task.

Robotics and AI technologies often have been perceived as radically innovative technologies that likely will cause problematic disruptions of society. In particular, ethical concerns about the simulation of high-level human capacities in social robots have engendered calls for various forms of ‘top-down’ regulation, guided by pre-set values and standards. Working from insights of a case study with mobile robots, we suggest in this discussion paper that we might do better in pursuing responsible innovation bottom-up: Moving among robots with minimal social interaction skills, citizens may practically acquire the technological knowledge that can protect them against anthropomorphizing overinterpretations and inappropriate attachments. We unfold this suggestion in three steps: First, reporting on our case study, we describe the limited role of value considerations among the design goals that currently guide developers and early adopters of mobile robots in the retail industry. Second, based on an observation from an ethnographic pilot study and related HRI research on mobile robots, we derive a proposal for responsible innovation bottom-up, as supervised social experiment and cultural learning. Third, we discuss the proposal, pro and contra.

PurposeAccurate 6D object pose estimation is essential for various robotic tasks. Uncertain pose estimates can lead to task failures; however, a certain degree of error in the pose estimates is often acceptable. This paper aims to enable the robots to make informed decisions by quantifying errors in the object pose estimate and acceptable errors for task success.Design/methodology/approachIn this paper, the authors introduce a framework for evaluating robotic task success under object pose uncertainty, representing both the estimated error space of the object pose and the acceptable error space for task success using multi-modal non-parametric probability distributions. The proposed framework pre-computes the acceptable error space for task success using dynamic simulations and subsequently integrates the pre-computed acceptable error space over the estimated error space of the object pose to predict the likelihood of the task succes.FindingsThe authors evaluated the proposed framework on two mobile manipulation tasks. Their results show that by representing the estimated and the acceptable error space using multi-modal non-parametric distributions, the authors achieve higher task success rates and fewer failures.Research limitations/implicationsTheir proposed framework is generic and can be applied to a wide range of robotic tasks requiring object pose estimation. Hence, given the recent advancements in object pose uncertainty estimation and dynamic simulations, the proposed framework, in conjunction with these advancements, has the potential to enable robots to make reliable and informed decisions under pose uncertainty.Originality/valueUnlike related works that model both acceptable error space and estimated error space using parametric uni-modal distributions, the authors model them as multi-modal distributions which is often the case in the real world.

In human-robot collaboration scenarios, mutual adaptation between the human and robot must occur to ensure high task performance. This requires robotic systems to be capable of reasoning based on a long-term history of interactions. In this paper, we present and evaluate a robot simulation system that facilitates adaptive robot behavior using ontology-based reasoning and behavior trees in an interactive robotic scanning task. A study with 38 participants compares our adaptive system with a static system in team performance and perceived system usability. Our results suggest that use of the adaptive system significantly reduced session time, leading users to perform the task 19.5% faster. Furthermore, participants reported significantly lower fatigue levels, while maintaining the same task performance as those using the static system.

The automation of robotic tasks requires high precision and adaptability, particularly in force-based operations such as insertions. Traditional learning-based approaches either rely on static datasets, which limit their ability to generalize, or require frequent manual intervention to maintain good performances. As a result, ensuring long-term reliability without human supervision remains a significant challenge. To address this, we propose an adaptive self-supervised learning framework for insertion classification that continuously improves its precision over time. The framework operates in real-time, incrementally refining its classification decisions by integrating newly acquired force data. Unlike conventional methods, it does not rely on pre-collected datasets but instead evolves dynamically with each task execution. Through real-world experiments, we demonstrate how the system progressively reduces execution time while maintaining near-perfect precision as more samples are processed. This adaptability ensures long-term reliability in force-based robotic tasks while minimizing the need for manual intervention.

In Mobile Manipulation (MM), navigation and manipulation are generally solved as subsequent disjoint tasks. Combined optimization of navigation and manipulation costs can improve the time efficiency of MM. However, this is challenging as precise object pose estimates, which are necessary for such combined optimization, are often not available until the later stages of MM. Moreover, optimizing navigation and manipulation costs with conventional planning methods using uncertain object pose estimates can lead to failures and hence requires replanning. Hence, in the presence of object pose uncertainty, preactive approaches are preferred. We propose such a pre-active approach for determining the base pose and pre-grasp manipulator configuration to improve the time efficiency of MM. We devise a Reinforcement Learning (RL) based solution that learns suitable base poses for grasping and pre-grasp manipulator configurations using layered learning that guides exploration and enables sample-efficient learning. Further, we accelerate learning of pre-grasp manipulator configurations by providing dense rewards using a predictor network trained on previously learned base poses for grasping. Our experiments validate that in the presence of uncertain object pose estimates, the proposed approach results in reduced execution time. Finally, we show that our policy learned in simulation can be easily transferred to a real robot. The code repository and the supplementary video can be found on the project webpage*.

The transition from robots and humans working separately to performing tasks in a shared workspace presents new challenges, such as adapting the robot’s behavior to the state of the human worker. For instance, maintaining an appropriate state of human-to-robot trust is a central antecedent of effective human-robot collaboration. To explore this area, we present a study investigating the effect of trust calibration on team performance. We investigated with 38 participants the effects of using robot apologies as well as two different framing scenarios in a robotic Pick and Place task.

We explore an alternative approach to the design of robots that deviates from the common envisionment of having one unified agent. What if robots are depicted as an agentic ensemble where agency is distributed over different components? In the project presented here, we investigate the potential contributions of this approach to creating entertaining and joyful human-robot interaction (HRI), which also remains comprehensible to human observers. We built a service robot—which takes care of plants as a Plant-Watering Robot (PWR)—that appears as a small ship controlled by a robotic captain accompanied by kinetic elements. The goal of this narrative design, which utilizes a distributed agency approach, is to make the robot entertaining to watch and foster its acceptance. We discuss the robot’s design rationale and present observations from an exploratory study in two contrastive settings, on a university campus and in a care home for people with dementia, using a qualitative video-based approach for analysis. Our observations indicate that such a design has potential regarding the attraction, acceptance, and joyfulness it can evoke. We discuss aspects of this design approach regarding the field of elderly care, limitations of our study, and identify potential fields of use and further scopes for studies.

In many applications, a mobile manipulator robot is required to grasp a set of objects distributed in space. This may not be feasible from a single base pose and the robot must plan the sequence of base poses for grasping all objects, minimizing the total navigation and grasping time. This is a Combinatorial Optimization problem that can be solved using exact methods, which provide optimal solutions but are computationally expensive, or approximate methods, which offer computationally efficient but sub-optimal solutions. Recent studies have shown that learning-based methods can solve Combinatorial Optimization problems, providing near-optimal and computationally efficient solutions. In this work, we present BASENET - a learning-based approach to plan the sequence of base poses for the robot to grasp all the objects in the scene. We propose a Reinforcement Learning based solution that learns the base poses for grasping individual objects and the sequence in which the objects should be grasped to minimize the total navigation and grasping costs using Layered Learning. As the problem has a varying number of states and actions, we represent states and actions as a graph and use Graph Neural Networks for learning. We show that the proposed method can produce comparable solutions to exact and approximate methods with significantly less computation time. The code and Reinforcement Learning environments will be made available on the project webpage*.

We present the concept and technical realisation for a cup that moves and lights up to bring itself to the attention of a person to trigger him/her taking a sip as a response. We then reflect on different ethical dimensions connected to the application of the cup in the context of people affected by dementia and describe first tests performed in elderly care homes. The concept is aimed at people with dementia in home or resident care who still have the ability to act, but tend to mentally drift away and thus require external impulses and triggers to drink. We found out that a substantial part of the residents fulfil these conditions. The cup moves and lights up in regular intervals if it has not been picked up recently. Once it is emptied, it alerts a caregiver to refill. Moreover, the degree or level of movement and light can be configured, depending on the person’s needs and reactions. This paper describes the core idea and the technical aspects of building the prototype. Finally, primary tests were conducted with the aim to construct a protocol and structure for an extended quantitative study.

Due to demographic change, health and elderly care systems are facing a shortage of qualified caregivers. This issue can be addressed by introducing welfare robots into people's homes, hospitals, and care institutions. To provide useful support, such robots must adapt to individual users and smoothly interact with them. From this perspective, we present advances on the development of proactive control for online individual user adaptation in a welfare robot guidance scenario, with the integration of three main modules: 1) navigation control; 2) visual human detection; and 3) temporal error correlation-based neural learning. The proposed control approach can drive a mobile robot to autonomously navigate in relevant indoor environments. At the same time, it can predict human walking speed based on visual information without prior knowledge of personality and preferences (i.e., walking speed). The robot then uses this prediction to continuously adapt its speed to individual users in a proactive online manner. We validate the performance of the proposed proactive robot control in different real-world environments with various users, including an elderly resident of a Danish elderly care center. The results show that the robot successfully and smoothly guided various users of different ages and average walking speeds (e.g., 0.2 m/s, 0.7 m/s, and 1.1 m/s) to target locations over distances of 25–60 m. All in all, this study captures a wide range of research from robot control technology development to technological validity in a relevant environment and system prototype demonstration in an operational environment (i.e., an elderly care center).

Care of ageing adults has become a dominant field of application for assistive robot technologies, promising support for ageing adults residing in care homes and staff, in dealing with practical routine tasks and providing social and emotional relieve. A time consuming and human intensive necessity is the maintenance of high hygiene quality in care homes. Robotic vacuum cleaners have been proven effective for doing the job elsewhere, but-in the context of care homes-are counterproductive for residents' well-being and do not get accepted. This is because people with dementia manifest their agency in more implicit and emotional ways, while making sense of the world around them. Starting from these premises, we explored how a zoomorphic designed vacuum cleaner could better accommodate the sensemaking of people with dementia. Our design reconceptualises robotic vacuum cleaners as a cat-like robot, referring to a playful behaviour and appearance to communicate a non-threatening and familiar role model. Data from an observational study shows that residents responded positively to our prototype, as most of them engaged playfully with it as if it was a pet or a cat-like toy, for example luring it with gestures. Some residents simply ignored the robot, indicating that it was not perceived as frightening or annoying. The level of activity influenced reactions; residents ignored our prototype if busy with other occupations, which proves that it did not cause significant disturbance. We further report results from focus group sessions with formal and informal caregivers who discussed a video prototype of our robot. Caregivers encouraged us to enhance the animal like characteristics (in behaviour and materiality) even further to result in richer interactions and provoke haptic pleasure but also pointed out that residents should not mistake the robot for a real cat.

Most of people's communication happens through body language and gestures. Gesture recognition in human-robot interaction is an unsolved problem which limits the possible communication between humans and robots in today's applications. Gesture recognition can be considered as the same problem as action recognition which is largely solved by deep learning, however, current publicly available datasets do not contain many classes relevant to human-robot interaction. In order to address the problem, a human-robot interaction gesture dataset is therefore required. In this paper, we introduce HRI-Gestures, which includes 13600 instances of RGB and depth image sequences, and joint position files. A state of the art action recognition network is trained on relevant subsets of the dataset and achieve upwards of 96.9% accuracy. However, as the network is designed for the large-scale NTU RGB+D dataset, subpar performance is achieved on the full HRI-Gestures dataset. Further enhancement of gesture recognition is possible by tailored algorithms or extension of the dataset.

The well-being of older people in care homes does not only rely on health and bodily needs but also includes spiritual or social needs. The presence of plants and distraction from everyday routines are two rarely addressed issues in this regard. Having those in mind, we developed the concept of the 'Plant Watering Robot' (PWR), a robotic device that has a double purpose: to water plants and serve as an attraction to observers thereby creating amusement. It is designed as a little ship inhabited by a small 'captain' that is displayed as being in charge of the device's actions. The pilot interacts with various synchronized elements building up a narrative of being in charge of watering the plants. We first report on related work before describing the interaction concept in more detail. We then elaborate the technical implementation of the PWR focussing on mechanical and software aspects.

We present the concept and technical realisation for a cup that moves and lights up so as to bring itself to the attention of a person with dementia, to trigger taking a sip as a response. The concept is aimed at people with dementia in home or resident care who still have the ability to act, but tend to mentally drift away and thus require external impulses and triggers. The cup moves and lights up in regular intervals if it has not been picked up recently. Once it is emptied, it alerts a caregiver to refill. Moreover, the degree or level of movement and light can be configured, depending on the person’s needs and reactions. This paper describes the core idea and focuses on the technical aspects of building a prototype on Technology Readiness Level (TRL) 3.

Mobile robots are becoming more and more ubiquitous in our everyday living environments. Therefore, it is very important that people can easily interpret what the robot’s intentions are. This is especially important when a robot is driving down a crowded corridor. It is essential for people in its vicinity to understand which way the robot wants to go next. To explore what signals are the best for conveying its intention to turn, we implemented three lighting schemes and tested them out in an online experiment. We found that signals resembling automotive signaling work the best also for logistic mobile robots. We further find that people’s opinion of these signaling methods will be influenced by their demographic background (gender, age).