Human involvement in space missions is inherently limited by safety concerns and extreme environmental conditions. Therefore, exploration and infrastructure maintenance in space require the teleoperation of robotic systems with various capabilities. To support the user in achieving teleoperation goals under challenging conditions, such as limited bandwidth and communication latency, a multimodal user interface (UI) and a scalable autonomy commanding concept are needed. This approach allows the operator to command the robot using different input modalities, including joystick and force-feedback devices for direct teleoperation, as well as a graphical user interface (GUI) for task-level commands. Complex robotic systems offer a wide variety of task-level commands, making it difficult for the user to select the optimal command to achieve a given task goal. Enabling the robot to build a Shared Mental Model (SMM) with the operator helps to overcome this challenge. An SMM refers to a shared and compatible mental representation that humans and robots hold as members of a team, encompassing information about the common task goal (task model) and the skills and intentions of team members (user model). SMMs are associated with improved task performance and reduced mental workload for the operator in human-robot teleoperation. Since the robot initially lacks knowledge about the task goal and cannot directly create an SMM, we design a framework that enables the robot to estimate the user's (sub-)task goal based on a priori knowledge of the user's task execution behavior. We integrate the a priori knowledge into a Bayesian estimation framework, combining it with information about user input and the robot's world state in the remote environment. Based on this, the robot estimates the user's subtask goal and determines the task-level command best suited to achieve it. This command is then recommended to the user. To evaluate the prediction accuracy of our framework, we use data from the International Space Station (ISS)-toground telerobotic experiments conducted in the Surface Avatar technology demonstration mission. We compare the task-level commands selected by the user with the predictions of our framework, as well as with a sequence of task-level commands for efficiently achieving the subtask goal that an expert user would choose. Furthermore, we analyze how such an SMM can support learning conduciveness and offer an outlook on how it may contribute to mutual learning in a fully sociotechnical system.

Future human spaceflight beyond low Earth orbit (LEO) will require robotic assistance to reduce astronaut workload while enhancing research capabilities. To advance humanoid service and assistance robotics for future exploration missions, the Surface Avatar ISS telerobotic technology demonstration mission, a collaboration between the German Aerospace Center (DLR) and the European Space Agency (ESA), demonstrates such concepts by enabling astronauts aboard the International Space Station (ISS) to remotely control heterogeneous robots in a Martian analog environment. Reliable localization and mapping are critical for autonomous capabilities in environments where established methods, such as the Global Navigation Satellite System (GNSS), are unavailable or unreliable. This paper presents a visual simultaneous localization and mapping (SLAM) framework designed for diverse robotic platforms, including DLR's humanoid Rollin' Justin, quadruped Bert, and ESA's Interact rover. DLR's stationary Enhanced LAnder Functional AssistaNT (ELAFANT) and ESA's Spot quadruped, relying on a proprietary navigation stack, are excluded from the study. Building on Rollin' Justin's initial implementation, the framework was extended to robots with limited sensing capabilities using a camera-agnostic SLAM pipeline. A unified mapping approach enables consistent localization across platforms and incremental map extension, supporting real-time performance in large-scale environments. The implementation was validated in six orbit-to-ground telerobotic sessions, confirming reliable performance and scalability. Results from these sessions demonstrate the robustness of the approach and its ability to maintain reliable localization across multiple robots in a challenging, GNSS-denied setting.

Handling raw products in supermarkets is still nowadays a typically manual task. First steps toward automation of these actions are being taken, typically for large production volumes with tailored solutions in well controlled environment. This paper presents a parallel mechanism robotic gripper capable of gripping by scooping and caging deformable fresh-produce like fish and meat as part of an automated logistic delivery chain with high mix low volume. The design motivation has been derived from human pick and place strategies to achieve flexible and robust capabilities in an unstructured environment. The gripper is designed using a variable stiffness actuator, to provide the delicacy required to handle the products without damaging them. Human experiments also provide input to a grasp planning framework to pick the best suited grasp strategy and position for a given object in a given surrounding. Finally, extensive testing of the gripper and grasping framework in lab and industrial conditions demonstrate its suitability for production in the real world setting.

With increasing task complexity in space robotic mission designs, advancement in the command technology and remote operation of the robotic team becomes a key challenge for providing an effective human-robotic team interface to help facilitate successful task execution. In the International Space Station (ISS)-to-ground telerobotic experiments, Surface Avatar led by DLR with partner ESA, a team of robots on the (Earth) surface is commanded by an astronaut in orbit to perform various tasks in a simulated space habitat. The ISS crew can select different input modalities to command the robots with direct teleoperation with the aid of force reflection in some instances, which provides great user immersion and interaction with the environment. At the other end of the Scalable Autonomy spectrum, the robot team can also be commanded at the task level, utilizing the robot's local intelligence to plan and execute at the crew's high-level direction. By scaling and mixing these telerobotic command capabilities, we can bring human and robot intelligence together to solve and execute more complex and unknown tasks. The teleoperator, whether an astronaut or ground-based expert, would be able to effectively command a surface robotic team in a wide variety of conditions. A concern raised for autonomously executed robotic task is the ability to cope with unexpected situations in a timely manner. To help enhance this command scalability, particularly for a more seamless transition between the command modes, the current research presented in this paper introduces a system that allows the crew to intervene with direct user input to provide course correction during autonomous task execution mid-task execution. This work explores different solutions to provide the human teleoperator with situational awareness of the robot's action, and approaches for user intervention. One approach we developed provides guidance forces based on the autonomous task and uses the movement of the haptic device as a comparison metric for interpreting the user intention. The user can observe the robot's action via the video stream, and let the robot complete the task autonomously. The robot executes the tasks autonomously as long as the robot's planned action (e.g. motion) concurs with user's intention. Should the need or desire arise, the user can take over with direct teleoperation at any moment using the haptic interface. The haptic cues provided by the haptic input device can take different forms such as pose, position, or velocity. In order to understand the effectiveness of these different approaches for allowing the teleoperator to take over the robot's autonomous task, particularly for space deployment, an implementation for user intervention is tested during the Surface Avatar ISS-Earth telerobotic experiment in 2025 where the ISS crew commanded a team of robots to perform various sample handling tasks with Scalable Autonomy teleoperation. Its outcome and astronaut feedback are discussed. Furthermore, a user study is carried out on-ground in the same simulated space habitat with the time-delay conditions experienced by the ISS crew to study the performance and usability of seamless switching of telecommand modalities in Scalable Autonomy teleoperation.

Generating robust and reactive manipulation strategies that can adapt to changing context information is a challenging task in robotics. Over the years, Learning from Demonstration (LfD) has emerged as an intuitive and effective solution for generating reactive policies, particularly by following dynamical-system(DS)-based approaches. However, most state-of-the-art DS-based approaches focus on addressing the robustness limitations, overlooking the modulation of policies in response to the environment. As a result, they tend to be inflexible with respect to parameterization by task-dependent variables. In this work, we build on existing work on policy fusion and uncertainty quantification to propose a context-adaptive policy framework that combines task-parameterized, robust and reactive manipulation. For this, we use LfD to acquire a policy that is conditioned on the robot state and low-dimensional task-dependent parameters reflecting the environment. We combine the learned policy with additional uncertainty-aware policies using a Mixture of Experts (MoE) formulation to improve its out-of-distribution (OOD) robustness and convergence behavior. The approach is evaluated on the LASA handwriting dataset and on a real 7-DoF robot in three scenarios: force-conditioned grasping, manipulation of deformable food items and object-centric grasping.

With each space robotic mission, a team of specialists on ground is usually required to support the robot throughout its mission. When scaling from single robots to fleets of robots, this will no longer be a viable way of conducting missions. Instead, in order to operate more autonomously in complex scenarios and adapt to dynamically changing environments without relying on a team of human experts, robots must be able to create a mental model of their surroundings. The relevance of this requirement becomes especially salient in the Surface Avatar ISS-to-ground telerobotic technology demonstration mission, where astronauts onboard the International Space Station command a team of heterogeneous robots in our lab. This work focuses on the final Surface Avatar session with NASA astronaut Jonny Kim where he was tasked with commanding the robots in a collaborative manner in order to complete simulated science and exploration tasks. To equip a complex robot such as DLR's Rollin' Justin with the ability to collaborate and coordinate with other robots, we deploy a combination of models and heuristics that allow it to create a mental model of its environment including its surrounding agents. Rollin' Justin also shares the estimated states generated in the mental model with the other robots. In this paper, we describe the application of this concept to our space experiment, present the models used to create the mental model, and evaluate it based on data collected in the final Surface Avatar experiment.

The German Aerospace Center (DLR), as part of the International Space Exploration Coordination Group (ISECG), shares the vision of sustainable human and robotic exploration of the Solar System. In this context, the EDEN LUNA project introduces a Moon-analogue greenhouse facility for the demonstration of nearly closed-loop bio-regenerative life support systems technology and plants cultivation for the purpose of feeding a crew. An autonomous robotic system EDEN Versatile End-effector (EVE) is to be integrated into the EDEN LUNA greenhouse, to partner with humans in support of this food production and to enable sustained extra-terrestrial exploration. EVE operates in a shared-autonomy manner, wherein an operator issues commands which trigger autonomous operation of robotic system. This is a highly significant feature which directly impacts the workload of astronauts inside the greenhouse. This preliminary study describes the design of EVE and compares EVE’s preliminary performance to existing studies on agricultural robotics. It also investigates space plant cultivation experiments and ground-based green-house analogues to compare them with the automatized scenario presented in this work.

Robots are crucial for exploring distant celestial bodies. The Surface Avatar ISS-to-Earth Telerobotic Technology Demonstration Mission investigates how to command a heterogeneous team of robots from orbit using scalable autonomy. In this experiment series astronauts aboard the International Space Station (ISS) command a team of robots located on Earth. One challenge identified in these experiments is that the robot’s decision-making process is affected by environmental uncertainty especially for the robot’s pose estimation and navigation. Traditional planning algorithms assume perfect knowledge of the robot’s surroundings, neglecting the role of imperfect sensing. This paper addresses this limitation by developing an uncertainty-aware planning method and demonstrating its application to address perception inaccuracies in navigation, paving the way for its broader adoption in other domains. The proposed uncertainty aware planning framework is tested for localization and navigation by the humanoid robot Rollin’ Justin during a space-to-ground telerobotic experiment as part of the Surface Avatar mission.

Team EDAN and pilot Mattias Atzenhofer won the first Assistance Robot Race at Cybathlon 2024.

With continuous advancements in robotics, both in hardware and in software, the feasibility to deploy robotic assistants as co-workers for astronauts in real mission scenarios is coming in sight. In the context of the Surface Avatar International Space Station (ISS) telerobotic technology demonstration mission, we study the requirements in terms of user interface (UI), robotic capabilities, and communication to enable efficient usage of robots as astronaut's co-workers. During the experiments, astronauts onboard the ISS command a team of heterogeneous robots at the German Aerospace Center (DLR) in Oberpfaffenhofen, Germany, to perform experimental tasks in a Mars analog environment. In order to complete the tasks successfully, the astronauts have to select between different robot command modalities, namely teleoperation and supervised autonomy. While previous publications have mostly focused on the UI, the interfaces between robots and the UI, and the overall mission concept, this work sheds light on the robotic back-end and provides a description of our reference implementation. Utilizing the humanoid robot Rollin’ Justin as our prime use case, we describe the modules that enable the robotic capabilities that are offered to the astronauts as well as their implementations. As a core aspect of Surface Avatar is the ability to select from different command modes, i.e. supervised autonomy and direct teleoperation, we put special focus on the high-level modules that enable supervised autonomy, such as knowledge representation, belief state representation, and reasoning, as well as the teleoperation interfaces. The paper also describes the integration of the aforementioned modules into the overall system. In this work we share the decisions and iteration processes that lead up to our current design, the motivation behind the decisions, the limitations they imply on the system, and the lessons learned during the process. This work particular examines these modules in the context of the Surface Avatar experiment session and describes, in particular, the improvements that have been achieved in comparison to previous versions. While the system continues to evolve to support new features for upcoming experiment sessions, our description covers the state of the robot during the first ISS experiment sessions and the two following prime sessions of Surface Avatar.

Robotics will continue to be essential for future space missions, supporting exploration, construction, and maintenance on distant celestial bodies. However, instructing these systems from Earth is challenging because of communication delays and environmental uncertainties. A more suitable approach involves commanding robots from orbit. This setup reduces the time delay between operator and robot allowing for more detailed robot commands up the way to direct teleoperation. The Surface Avatar experiment explores mission scenarios of human -robot team collaboration, where communication time delays are under several seconds. This allows for significantly more complex tasks with human -in -the -loop execution. In our mission design, a team of heterogeneous robots on Earth is controlled from the International Space Station (ISS) using Scalable Autonomy, balancing autonomous execution with human intervention. In this paper, we propose a recovery method where robots autonomously detect issues and request astronaut or ground control support to address task failures. This frees the astronaut from constant supervision. One way to use the newly gained time is to switch focus on commanding another robot. Our framework is validated in a Martian surface mockup environment during an ISS-to-earth experiment session, demonstrating its effectiveness for future missions.

We introduce NealAI, the first AI chat assistant to support astronauts with question answering during a space telerobotics experiment. In the Surface Avatar mission, an ISS crew member controlled a heterogeneous team of four robots in a simulated Martian environment. NealAI uses a Retrieval-Augmented Generation (RAG) approach, enabling a Large Language Model (LLM) to dynamically retrieve relevant context about the experiment and its robots, and deliver accurate, context-aware responses. To adhere to privacy requirements and computational costs, NealAI is based on a single small-scale LLM running locally. We assessed NealAI’s performance in different evaluations, including a preliminary experiment with an ISS crew member teleoperating the robots, as well as a set of offline tests to evaluate the LLM context selection, the response correctness, and when (and why) hallucinations occur. Results demonstrate the feasibility and limitations of using a small-scale LLM on a RAG-based chat assistant during a space telerobotic experiment. Finally, we report some conclusions and lessons learned.

Uncertainties in contact dynamics and object geometry remain significant barriers to robust robotic manipulation. Caging helps mitigate these uncertainties by constraining an object's mobility without requiring precise contact modeling. Existing caging research often treats morphology and policy optimization as separate problems, overlooking their synergy. In this paper, we introduce CageCoOpt, a hierarchical framework that jointly optimizes manipulator morphology and control policy for robust caging-based manipulation. The framework employs reinforcement learning for policy optimization at the lower level and multi-task Bayesian optimization for morphology optimization at the upper level. We incorporate a caging metric into both optimization levels to encourage caging configurations and thereby improve manipulation robustness. The evaluation consists of four manipulation tasks and demonstrates that co-optimizing morphology and policy improves task performance under uncertainties, establishing caging-guided co-optimization as a viable approach for robust manipulation.

Dexterity and strength are essential for performing a variety of tasks in the unstructured environments of household services and craftsmanship. For these tasks, we have developed neoDavid, a robust humanoid robot with dexterous manipulation skills. neoDavid has joints with variable-stiffness actuators (VSAs), which have mechanically adjustable elasticity in the drive train, a continuum elastic neck, and a gravitationally compensated torso with overload couplings. We present our modular approach in the development, starting from system architecture over mechatronic components, communication, and control to higher-level software. We demonstrate how this modularity enables scalable enhancements, allowing us to evolve the system from a single arm and hand into a complete humanoid robot. Additionally, we highlight advancements in perception, manipulation, and motion planning, along with the implementation of offline task planning utilizing capability maps. The versatile character in terms of dexterity and robustness is demonstrated in challenging applications, e.g., handling a drill hammer, fine manipulation of a pipette, and emptying a dishwasher.

Logistics and service operations involving parcel preparation, delivery, and unpacking from a supply point to a user's home could be carried out completely by robots in the near future, taking advantage of the capabilities of the different robot morphologies for the logistics, outdoor, and domestic environments. The use of robots for parcel delivery can contribute to the goals of sustainability and reduced emissions by exploiting their different locomotion modalities (wheeled, legged, and aerial). This article reports the development and results obtained from the first robotics hackathon celebrated as part of the European Robotics and Artificial Intelligence Network involving eight robotic platforms in three domains: 1) an industrial robotic arm for parcel preparation at the supply point, 2) a Centauro robot, a dual-arm aerial manipulator, and a wheeled-legged quadruped for parcel transportation, and 3) two humanoid robots and two commercial mobile manipulators for parcel delivery and unpacking in domestic scenarios. The article describes the joint operation and the evaluation scenario, the features and capabilities of the robots, particularly those involved in the realization of the tasks, and the lessons learned.

Humans exhibit a particular compliant behavior in interactions with their environment. Facilitated by fast physical reasoning, humans are able to rapidly alter their compliance, enhancing robustness and safety in active environments. Transferring these capabilities to robotics is of utmost importance particularly as major space agencies begin investigating the potential of cooperative robotic teams in space. In this scenario, robots in orbit or on planetary surfaces are meant to support astronauts in exploration, maintenance, and habitat building to reduce costs and risks of space missions. A major challenge for interactive robot teams is establishing the capability to act in and interact with dynamic environments. Analogous to humans, the robot should be not only particularly compliant in case of unexpected collisions with other systems but also able to cooperatively handle objects requiring accurate pose estimation and fast trajectory planning. Here, we show that these challenges can be attenuated through an enhancement of active robot compliance introducing a virtual plastic first-order impedance component. We present how elasto-plastic compliance can be realized via energy-based detection of active environments and how evasive motions can be enabled through adaptive plastic compliance. Two space teleoperation experiments using different robotic assets confirm the potential of the method to enhance robustness in interaction with articulated objects and facilitate robot cooperation. An experiment in a health care facility presents how the same method analogously solidifies robotic interactions in human-robot shared environments by giving the robot a subordinate role.

Spacefaring nations have already expressed their plans for a sustainable human and robotic exploration on the Moon. This endeavor highlighted in the Global Exploration Roadmap (GER) foresees the development of infrastructures such as habitats, greenhouses, science labs, power plants, and mining facilities. Following this long-term vision, the German Aerospace Center (DLR) EDEN LUNA Project presents a Moon-analogue greenhouse facility which can demonstrate nearly closed-loop bio-regenerative life support systems technology and aim to produce fresh food for astronauts on the Moon in the near future. To optimize the food production and overcome challenges inherent to space missions, the EDEN Versatile End-effector (EVE) is integrated to the EDEN LUNA Greenhouse. This support system is a valuable payload which will automatize the tasks of the entire plant cultivation process: from germination to harvesting. The automatization is particularly relevant when the food production is intensified either seasonally or in a future scaled-up scenario. The EVE system encompasses a linear rail system installed on the ceiling of the greenhouse, a 7-Degrees of Freedom (DOF) autonomous robotic arm with high precision joint configuration, a sensorized robotic hand which can grasp delicate objects, and a sophisticated computer vision camera with plant monitoring capabilities. When in operation, the EVE system uses shared autonomy features. Thus, while it maintains the human in the loop for some of the decision-making processes, it can also function with some level of autonomy. A set of tasks previously defined by an astronaut in the end of an operational day and carried out autonomously during the night by the EVE system is one example of this human-robot collaboration. In addition, an optimized motion planning will ensure that the EVE system can perform constrained manipulation tasks in a limited workspace observing energy efficiency and safety requirements. This is explained in the paper with the abstraction of the different robotic control levels which range from the high-level view for non-experts in robotics to the motion planning level and their interconnections. The EVE system is currently in development at the DLR Robotic and Mechatronics Center (RMC) in Oberpfaffenhofen. In 2024, it will be integrated to the EDEN LUNA Greenhouse at the DLR Institute of Space Systems in Bremen. Finally, by the end of 2025, it will start operations in the ESA/DLR LUNA facility at the European Astronaut Centre (EAC) in Cologne.

For certain tasks in logistics, especially bin picking and packing, humans resort to a strategy of grasping multiple objects simultaneously, thus reducing picking and transport time. In contrast, robotic systems mainly grasp only one object per picking action, which leads to inefficiencies that could be solved with a smarter gripping hardware and strategies. Development of new manipulators, robotic hands, hybrid or specialized grippers, can already consider such challenges for multi-object grasping in the design stages. This paper introduces different hardware solutions and tests possible grasp strategies for the simultaneous grasping of multiple objects (SGMO). The four hardware solutions presented here are: an under-actuated Constriction Gripper, Linear Scoop Gripper suitable for deform-able object grasping, Hybrid Compliant Gripper equipped with mini vacuum gripper on each fingertip, and a Two-finger Palm Hand with fingers optimized by simulation in pybullet for maximum in-hand manipulation workspace. Most of these hardware solutions are based on the DLR CLASH end-effector and have variable stiffness actuation, high impact robustness, small contact forces, and low-cost design. For the comparison of the capability to simultaneously grasp multiple objects and the capability to grasp a single delicate object in a cluttered environment, the manipulators are tested with four different objects in an extra designed benchmark. The results serve as guideline for future commercial applications of these strategies.