Human involvement in space missions is inherently limited by safety concerns and extreme environmental conditions. Therefore, exploration and infrastructure maintenance in space require the teleoperation of robotic systems with various capabilities. To support the user in achieving teleoperation goals under challenging conditions, such as limited bandwidth and communication latency, a multimodal user interface (UI) and a scalable autonomy commanding concept are needed. This approach allows the operator to command the robot using different input modalities, including joystick and force-feedback devices for direct teleoperation, as well as a graphical user interface (GUI) for task-level commands. Complex robotic systems offer a wide variety of task-level commands, making it difficult for the user to select the optimal command to achieve a given task goal. Enabling the robot to build a Shared Mental Model (SMM) with the operator helps to overcome this challenge. An SMM refers to a shared and compatible mental representation that humans and robots hold as members of a team, encompassing information about the common task goal (task model) and the skills and intentions of team members (user model). SMMs are associated with improved task performance and reduced mental workload for the operator in human-robot teleoperation. Since the robot initially lacks knowledge about the task goal and cannot directly create an SMM, we design a framework that enables the robot to estimate the user's (sub-)task goal based on a priori knowledge of the user's task execution behavior. We integrate the a priori knowledge into a Bayesian estimation framework, combining it with information about user input and the robot's world state in the remote environment. Based on this, the robot estimates the user's subtask goal and determines the task-level command best suited to achieve it. This command is then recommended to the user. To evaluate the prediction accuracy of our framework, we use data from the International Space Station (ISS)-toground telerobotic experiments conducted in the Surface Avatar technology demonstration mission. We compare the task-level commands selected by the user with the predictions of our framework, as well as with a sequence of task-level commands for efficiently achieving the subtask goal that an expert user would choose. Furthermore, we analyze how such an SMM can support learning conduciveness and offer an outlook on how it may contribute to mutual learning in a fully sociotechnical system.

Future human spaceflight beyond low Earth orbit (LEO) will require robotic assistance to reduce astronaut workload while enhancing research capabilities. To advance humanoid service and assistance robotics for future exploration missions, the Surface Avatar ISS telerobotic technology demonstration mission, a collaboration between the German Aerospace Center (DLR) and the European Space Agency (ESA), demonstrates such concepts by enabling astronauts aboard the International Space Station (ISS) to remotely control heterogeneous robots in a Martian analog environment. Reliable localization and mapping are critical for autonomous capabilities in environments where established methods, such as the Global Navigation Satellite System (GNSS), are unavailable or unreliable. This paper presents a visual simultaneous localization and mapping (SLAM) framework designed for diverse robotic platforms, including DLR's humanoid Rollin' Justin, quadruped Bert, and ESA's Interact rover. DLR's stationary Enhanced LAnder Functional AssistaNT (ELAFANT) and ESA's Spot quadruped, relying on a proprietary navigation stack, are excluded from the study. Building on Rollin' Justin's initial implementation, the framework was extended to robots with limited sensing capabilities using a camera-agnostic SLAM pipeline. A unified mapping approach enables consistent localization across platforms and incremental map extension, supporting real-time performance in large-scale environments. The implementation was validated in six orbit-to-ground telerobotic sessions, confirming reliable performance and scalability. Results from these sessions demonstrate the robustness of the approach and its ability to maintain reliable localization across multiple robots in a challenging, GNSS-denied setting.

With increasing task complexity in space robotic mission designs, advancement in the command technology and remote operation of the robotic team becomes a key challenge for providing an effective human-robotic team interface to help facilitate successful task execution. In the International Space Station (ISS)-to-ground telerobotic experiments, Surface Avatar led by DLR with partner ESA, a team of robots on the (Earth) surface is commanded by an astronaut in orbit to perform various tasks in a simulated space habitat. The ISS crew can select different input modalities to command the robots with direct teleoperation with the aid of force reflection in some instances, which provides great user immersion and interaction with the environment. At the other end of the Scalable Autonomy spectrum, the robot team can also be commanded at the task level, utilizing the robot's local intelligence to plan and execute at the crew's high-level direction. By scaling and mixing these telerobotic command capabilities, we can bring human and robot intelligence together to solve and execute more complex and unknown tasks. The teleoperator, whether an astronaut or ground-based expert, would be able to effectively command a surface robotic team in a wide variety of conditions. A concern raised for autonomously executed robotic task is the ability to cope with unexpected situations in a timely manner. To help enhance this command scalability, particularly for a more seamless transition between the command modes, the current research presented in this paper introduces a system that allows the crew to intervene with direct user input to provide course correction during autonomous task execution mid-task execution. This work explores different solutions to provide the human teleoperator with situational awareness of the robot's action, and approaches for user intervention. One approach we developed provides guidance forces based on the autonomous task and uses the movement of the haptic device as a comparison metric for interpreting the user intention. The user can observe the robot's action via the video stream, and let the robot complete the task autonomously. The robot executes the tasks autonomously as long as the robot's planned action (e.g. motion) concurs with user's intention. Should the need or desire arise, the user can take over with direct teleoperation at any moment using the haptic interface. The haptic cues provided by the haptic input device can take different forms such as pose, position, or velocity. In order to understand the effectiveness of these different approaches for allowing the teleoperator to take over the robot's autonomous task, particularly for space deployment, an implementation for user intervention is tested during the Surface Avatar ISS-Earth telerobotic experiment in 2025 where the ISS crew commanded a team of robots to perform various sample handling tasks with Scalable Autonomy teleoperation. Its outcome and astronaut feedback are discussed. Furthermore, a user study is carried out on-ground in the same simulated space habitat with the time-delay conditions experienced by the ISS crew to study the performance and usability of seamless switching of telecommand modalities in Scalable Autonomy teleoperation.

With each space robotic mission, a team of specialists on ground is usually required to support the robot throughout its mission. When scaling from single robots to fleets of robots, this will no longer be a viable way of conducting missions. Instead, in order to operate more autonomously in complex scenarios and adapt to dynamically changing environments without relying on a team of human experts, robots must be able to create a mental model of their surroundings. The relevance of this requirement becomes especially salient in the Surface Avatar ISS-to-ground telerobotic technology demonstration mission, where astronauts onboard the International Space Station command a team of heterogeneous robots in our lab. This work focuses on the final Surface Avatar session with NASA astronaut Jonny Kim where he was tasked with commanding the robots in a collaborative manner in order to complete simulated science and exploration tasks. To equip a complex robot such as DLR's Rollin' Justin with the ability to collaborate and coordinate with other robots, we deploy a combination of models and heuristics that allow it to create a mental model of its environment including its surrounding agents. Rollin' Justin also shares the estimated states generated in the mental model with the other robots. In this paper, we describe the application of this concept to our space experiment, present the models used to create the mental model, and evaluate it based on data collected in the final Surface Avatar experiment.

Robots are crucial for exploring distant celestial bodies. The Surface Avatar ISS-to-Earth Telerobotic Technology Demonstration Mission investigates how to command a heterogeneous team of robots from orbit using scalable autonomy. In this experiment series astronauts aboard the International Space Station (ISS) command a team of robots located on Earth. One challenge identified in these experiments is that the robot’s decision-making process is affected by environmental uncertainty especially for the robot’s pose estimation and navigation. Traditional planning algorithms assume perfect knowledge of the robot’s surroundings, neglecting the role of imperfect sensing. This paper addresses this limitation by developing an uncertainty-aware planning method and demonstrating its application to address perception inaccuracies in navigation, paving the way for its broader adoption in other domains. The proposed uncertainty aware planning framework is tested for localization and navigation by the humanoid robot Rollin’ Justin during a space-to-ground telerobotic experiment as part of the Surface Avatar mission.

With continuous advancements in robotics, both in hardware and in software, the feasibility to deploy robotic assistants as co-workers for astronauts in real mission scenarios is coming in sight. In the context of the Surface Avatar International Space Station (ISS) telerobotic technology demonstration mission, we study the requirements in terms of user interface (UI), robotic capabilities, and communication to enable efficient usage of robots as astronaut's co-workers. During the experiments, astronauts onboard the ISS command a team of heterogeneous robots at the German Aerospace Center (DLR) in Oberpfaffenhofen, Germany, to perform experimental tasks in a Mars analog environment. In order to complete the tasks successfully, the astronauts have to select between different robot command modalities, namely teleoperation and supervised autonomy. While previous publications have mostly focused on the UI, the interfaces between robots and the UI, and the overall mission concept, this work sheds light on the robotic back-end and provides a description of our reference implementation. Utilizing the humanoid robot Rollin’ Justin as our prime use case, we describe the modules that enable the robotic capabilities that are offered to the astronauts as well as their implementations. As a core aspect of Surface Avatar is the ability to select from different command modes, i.e. supervised autonomy and direct teleoperation, we put special focus on the high-level modules that enable supervised autonomy, such as knowledge representation, belief state representation, and reasoning, as well as the teleoperation interfaces. The paper also describes the integration of the aforementioned modules into the overall system. In this work we share the decisions and iteration processes that lead up to our current design, the motivation behind the decisions, the limitations they imply on the system, and the lessons learned during the process. This work particular examines these modules in the context of the Surface Avatar experiment session and describes, in particular, the improvements that have been achieved in comparison to previous versions. While the system continues to evolve to support new features for upcoming experiment sessions, our description covers the state of the robot during the first ISS experiment sessions and the two following prime sessions of Surface Avatar.

Robotics will continue to be essential for future space missions, supporting exploration, construction, and maintenance on distant celestial bodies. However, instructing these systems from Earth is challenging because of communication delays and environmental uncertainties. A more suitable approach involves commanding robots from orbit. This setup reduces the time delay between operator and robot allowing for more detailed robot commands up the way to direct teleoperation. The Surface Avatar experiment explores mission scenarios of human -robot team collaboration, where communication time delays are under several seconds. This allows for significantly more complex tasks with human -in -the -loop execution. In our mission design, a team of heterogeneous robots on Earth is controlled from the International Space Station (ISS) using Scalable Autonomy, balancing autonomous execution with human intervention. In this paper, we propose a recovery method where robots autonomously detect issues and request astronaut or ground control support to address task failures. This frees the astronaut from constant supervision. One way to use the newly gained time is to switch focus on commanding another robot. Our framework is validated in a Martian surface mockup environment during an ISS-to-earth experiment session, demonstrating its effectiveness for future missions.

Previous research in human-robot-interaction in teleoperation suggests that human-robot teams with a Shared Mental Model (SMM) are able to achieve better task performances and lower the mental workload of the user. To create a SMM, the robot must be able to estimate the task model of the user and incorporate knowledge about the user into its user model. Most existing concepts use probabilistic methods or symbolic reasoning frameworks, with predefined features for estimating the user’s teleoperation goal. They are therefore highly dependent on a-priori knowledge about the task, thus limiting their flexibility. Only very few approaches mention or explicitly take into account knowledge about the individual user and lack adaptability to human behavior and user preferences. Machine Learning (ML) based concepts, on the other hand, are more flexible to adapt their SMM during the user interaction, but require a lot of data points for precise estimation. In this paper, we propose a new approach to address these limitations by combining the advantages of symbolic and subsymbolic methods, and dividing our estimation of the user’s teleoperation goal into subgoals next to the overall task goal. Furthermore, by integrating the knowledge about the individual user and human decision-making into the prediction of the user’s task goal, we can improve SMM creation of human-robot teams for teleoperation. Our approach is applied to an example space exploration task in a planetary surface habitat by a telerobotic team. The SMM implementation is described and analyzed, followed by an outlook on future development and validation.

Employing a team of robots for space exploration offers benefits such as increased spatial coverage and optimized work distribution. However, limited bandwidth, communication delays and environmental uncertainties require the use of an intuitive multi-modal user interface (UI) to control these robotic assets. This interface enables the crew to command the robots with various modalities, including joysticks and force-reflection input devices for direct teleoperation, and an intuitive graphical user interface (GUI) for task-level command supervision. How-ever, during task executions, errors may occur and robots may fail to carry out the commanded tasks. In the absence of careful consideration for conveying error information to users, the GUI may display a message notifying the occurrence of an execution failure, providing little to no context leaving a non-expert user with unclear and uninformative details. This lack of sufficient information makes the management of the robotic assets less certain, which causes an increase of mental workload on the crew. To reduce the astronauts's cognitive load and ultimately enhance their situational awareness, it is crucial for robots to communicate the details of the encountered errors through text and visual infographs. In this paper, we propose a novel robot agnostic framework based on the OpenUSD standards to support error-related knowledge exchange from robot to astronaut. This concept was tested in the German Aerospace Center (DLR) - European Space Agency (ESA) space technology demonstration mission, Surface Avatar. Results from the International Space Station (ISS)-to-ground telerobotic experiments in July 2024 validated our approach.

We introduce NealAI, the first AI chat assistant to support astronauts with question answering during a space telerobotics experiment. In the Surface Avatar mission, an ISS crew member controlled a heterogeneous team of four robots in a simulated Martian environment. NealAI uses a Retrieval-Augmented Generation (RAG) approach, enabling a Large Language Model (LLM) to dynamically retrieve relevant context about the experiment and its robots, and deliver accurate, context-aware responses. To adhere to privacy requirements and computational costs, NealAI is based on a single small-scale LLM running locally. We assessed NealAI’s performance in different evaluations, including a preliminary experiment with an ISS crew member teleoperating the robots, as well as a set of offline tests to evaluate the LLM context selection, the response correctness, and when (and why) hallucinations occur. Results demonstrate the feasibility and limitations of using a small-scale LLM on a RAG-based chat assistant during a space telerobotic experiment. Finally, we report some conclusions and lessons learned.

Logistics and service operations involving parcel preparation, delivery, and unpacking from a supply point to a user's home could be carried out completely by robots in the near future, taking advantage of the capabilities of the different robot morphologies for the logistics, outdoor, and domestic environments. The use of robots for parcel delivery can contribute to the goals of sustainability and reduced emissions by exploiting their different locomotion modalities (wheeled, legged, and aerial). This article reports the development and results obtained from the first robotics hackathon celebrated as part of the European Robotics and Artificial Intelligence Network involving eight robotic platforms in three domains: 1) an industrial robotic arm for parcel preparation at the supply point, 2) a Centauro robot, a dual-arm aerial manipulator, and a wheeled-legged quadruped for parcel transportation, and 3) two humanoid robots and two commercial mobile manipulators for parcel delivery and unpacking in domestic scenarios. The article describes the joint operation and the evaluation scenario, the features and capabilities of the robots, particularly those involved in the realization of the tasks, and the lessons learned.

Humans exhibit a particular compliant behavior in interactions with their environment. Facilitated by fast physical reasoning, humans are able to rapidly alter their compliance, enhancing robustness and safety in active environments. Transferring these capabilities to robotics is of utmost importance particularly as major space agencies begin investigating the potential of cooperative robotic teams in space. In this scenario, robots in orbit or on planetary surfaces are meant to support astronauts in exploration, maintenance, and habitat building to reduce costs and risks of space missions. A major challenge for interactive robot teams is establishing the capability to act in and interact with dynamic environments. Analogous to humans, the robot should be not only particularly compliant in case of unexpected collisions with other systems but also able to cooperatively handle objects requiring accurate pose estimation and fast trajectory planning. Here, we show that these challenges can be attenuated through an enhancement of active robot compliance introducing a virtual plastic first-order impedance component. We present how elasto-plastic compliance can be realized via energy-based detection of active environments and how evasive motions can be enabled through adaptive plastic compliance. Two space teleoperation experiments using different robotic assets confirm the potential of the method to enhance robustness in interaction with articulated objects and facilitate robot cooperation. An experiment in a health care facility presents how the same method analogously solidifies robotic interactions in human-robot shared environments by giving the robot a subordinate role.

In uneven terrains of lunar and planetary surfaces, a teleoperated legged robot enables the exploration of areas that may be inaccessible to wheeled rovers. The on-going DLR-ESA Surface Avatar technology demonstration mission studies and validates technologies needed to realize the command of a heterogeneous robotic team with various command modalities. In the latest experiments, the crew on-board the International Space Station (ISS) is tasked to command a team of different robots with Scalable Autonomy, meaning they may choose to command the robots through direct control, shared control, or delegate tasks with Supervised Autonomy. One of the robots in the robotic team is DLR’s small quadruped, Bert. Equipped with a robotic arm on its back, also serving as camera mount, Bert can observe its surroundings and pick up small objects. Due to its serial-elastic joints, the hardware is robust to impacts, which is critical for space deployment. Applying adaptive learning strategies, Bert’s gaits can be (re)trained directly on hardware to adapt to different environment and gravity conditions. The quadruped as well as its back-mounted arm can be commanded through direct control by the astronauts via a joystick on-board the ISS. Additionally, simple preset task-level commands complement the user interface to ease the astronauts’ workload. This paper presents the ISS experiments of the first legged robot telecommanded from space. Examining different aspects of Bert’s telerobotic performance as well as the ISS crew feedback, we discuss the feasibility of teleoperated walking robots in space exploration.

This study investigates error handling intricacies in supervised autonomy orbit-to-ground teleoperation for space exploration robots, emphasizing scenarios with communication delays that render Earth-based ground control assistance unfeasible. In this setting, one major challenge lies in empowering the crew to independently mitigate robot errors that may occur as the robot plans its actions. To address this limitation of current supervised autonomy interfaces, we propose a third-person perspective and game design principles to improve environmental awareness in error situations. 16 experts with similar technical background as the target crew members tested the interface in a physical user study, while 42 people assessed it in an online study. We conclude that a third-person view brings significant improvements to mental workload, overall experience and the ability to identify and rectify planning errors.

Future crewed missions beyond low earth orbit will greatly rely on the support of robotic assistance platforms to perform inspection and manipulation of critical assets. This includes crew habitats, landing sites or assets for life support and operation. Maintenance and manipulation of a crewed site in extraterrestrial environments is a complex task and the system will have to face different challenges during operation. While most may be solved autonomously, in certain occasions human intervention will be required. The telerobotic demonstration mission, Surface Avatar, led by the German Aerospace Center (DLR), with partner European Space Agency (ESA), investigates different approaches offering astronauts on board the International Space Station (ISS) control of ground robots in representative scenarios, e.g. a Martian landing and exploration Site. In this work we present a feasibility study on how to integrate auditory information into the mentioned application. We will discuss methods for obtaining audio information and localizing audio sources in the environment, as well as fusing auditory and visual information to perform state estimation based on the gathered data. We demonstrate our work in different experiments to show the effectiveness of utilizing audio information, the results of spectral analysis of our mission assets, and how this information could help future astronauts to argue about the current mission situation.

Crewed missions to celestial bodies such as Moon and Mars are in the focus of an increasing number of space agencies. Precautions to ensure a safe landing of the crew on the extraterrestrial surface, as well as reliable infrastructure on the remote location, for bringing the crew back home are key considerations for mission planning. The European Space Agency (ESA) identified in its Terrae Novae 2030+ roadmap, that robots are needed as precursors and scouts to ensure the success of such missions. An important role these robots will play, is the support of the astronaut crew in orbit to carry out scientific work, and ultimately ensuring nominal operation of the support infrastructure for astronauts on the surface. The METERON SUPVIS Justin ISS experiments demonstrated that supervised autonomy robot command can be used for executing inspection, maintenance and installation tasks using a robotic co-worker on the planetary surface. The knowledge driven approach utilized in the experiments only reached its limits when situations arise that were not anticipated by the mission design. In deep space scenarios, the astronauts must be able to overcome these limitations. An approach towards more direct command of a robot was demonstrated in the METERON ANALOG-1 ISS experiment. In this technical demonstration, an astronaut used haptic telepresence to command a robotic avatar on the surface to execute sampling tasks. In this work, we propose a system that combines supervised autonomy and telepresence by extending the knowledge driven approach. The knowledge management is based on organizing the prior knowledge of the robot in an object-centered context. Action Templates are used to define the knowledge on the handling of the objects on a symbolic and geometric level. This robot-agnostic system can be used for supervisory command of any robotic coworker. By integrating the robot itself as an object into the object-centered domain, robot-specific skills and (tele-)operation modes can be injected into the existing knowledge management system by formulating respective Action Templates. In order to efficiently use advanced teleoperation modes, such as haptic telepresence, a variety of input devices are integrated into the proposed system. This work shows how the integration of these devices is realized in a way that is agnostic to the input devices and operation modes. The proposed system is evaluated in the Surface Avatar ISS experiment. This work shows how the system is integrated into a Robot Command Terminal featuring a 3-Degree-of-Freedom Joystick and a 7-Degree-of-Freedom haptic input device in the Columbus module of the ISS. In the preliminary experiment sessions of Surface Avatar, two astronauts on orbit took command of the humanoid service robot Rollin’ Justin in Germany. This work presents and discusses the results of these ISS-to-ground sessions and derives requirements for extending the scalable autonomy system for the use with a heterogeneous robotic team.

This paper proposes a knowledge-driven teleoperation framework that enables multiple operators to command a team of robots to execute complex tasks in an efficient and intuitive manner. The framework leverages a shared knowledge base that captures domain-specific information and procedural knowledge about the task at hand. This knowledge base is used by a hybrid planner to generate context-specifically relevant commands for supervised autonomy robot command as well as direct teleoperation modes. By filtering the available commands, the operators are guided in their decision-making towards efficient task completion. This paper further extends our knowledge driven approach to address the switching between multiple operators and robotic assets, with the aim to be able scale up human-robot team for space exploration. Overall, this work represents a step towards more intelligent and collaborative teleoperation systems. The described system will be used in the Surface Avatar ISS-to-ground experiments slated for July 2023.