Foundation models have demonstrated impressive capabilities across diverse domains, while imitation learning provides principled methods for robot skill adaptation from limited data. Combining these approaches holds significant promise for direct application to robotics, yet this combination has received limited attention, particularly for industrial deployment. We present a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture, maintaining a protective abstraction layer between the language model and robot hardware. Our approach leverages pre-trained LLMs to select and parameterize specific tools for adapting robot skills without requiring fine-tuning or direct model-to-robot interaction. We demonstrate the framework on a 7-DoF torque-controlled robot performing an industrial bearing ring insertion task, showing successful skill adaptation through natural language commands for speed adjustment, trajectory correction, and obstacle avoidance while maintaining safety, transparency, and interpretability.

Learning from demonstration (LfD) offers an intuitive alternative to manual coding by leveraging natural human behavior, while Human-Robot Interaction (HRI) provides an intuitive means to refine and adapt learned skills. Among interaction modalities, physical contact is a natural and effective way to convey intent. In order to leverage such modality, robots need to be able to distinguish physical contacts coming from deliberate human interactions (e.g. to correct a learned skill) from those from environmental factors (e.g. task-related). In this letter, we introduce a novel interactive framework for physically modulating learned robot skills. Our method builds on a state-of-the-art energy-tank-based intention detection approach to capture degree-of-freedom(DoF)-specific modulations and, accordingly, incorporate user-defined via-points into the learned skills. In contrast to common approaches, corrections are applied selectively to the relevant DoFs, preserving the original skill behavior in the remaining dimensions. Moreover, we leverage uncertainty in the demonstration data to modulate the tank dynamics, allowing users more or less time to intervene in regions of different uncertainty. We validate our approach on a torque-controlled 7-DoF robot executing a learned task of inserting a bearing ring, where physical human corrections are used to adapt to changes in the environment.

Industrial robot applications require increasingly flexible systems that non-expert users can easily adapt for varying tasks and environments. However, different adaptations benefit from different interaction modalities. We present an interactive framework that enables robot skill adaptation through three complementary modalities: kinesthetic touch for precise spatial corrections, natural language for high-level semantic modifications, and a graphical web interface for visualizing geometric relations and trajectories, inspecting and adjusting parameters, and editing via-points by drag-and-drop. The framework integrates five components: energy-based human-intention detection, a tool-based LLM architecture (where the LLM selects and parameterizes predefined functions rather than generating code) for safe natural language adaptation, Kernelized Movement Primitives (KMPs) for motion encoding, probabilistic Virtual Fixtures for guided demonstration recording, and ergodic control for surface finishing. We demonstrate that this tool-based LLM architecture generalizes skill adaptation from KMPs to ergodic control, enabling voice-commanded surface finishing. Validation on a 7-DoF torque-controlled robot at the Automatica 2025 trade fair demonstrates the practical applicability of our approach in industrial settings.

Lengthy setup processes that require robotics expertise remain a major barrier to deploying robots for tasks involving high product variability and small batch sizes. As a result, collaborative robots, despite their advanced sensing and control capabilities, are rarely used for surface finishing in small-scale craft and manufacturing settings. To address this gap, we propose a novel robot programming approach that enables non-experts to intuitively program robots through interactive, task-focused workflows. For that, we developed a new surface segmentation algorithm that incorporates human input to identify and refine workpiece regions for processing. Throughout the programming process, users receive continuous visual feedback on the robot's learned model, enabling them to iteratively refine the segmentation result. Based on the segmented surface model, a robot trajectory is generated to cover the desired processing area. We evaluated multiple interaction designs across two comprehensive user studies to derive an optimal interface that significantly reduces user workload, improves usability and enables effective task programming even for users with limited practical experience.

This paper presents a novel concept for a reconfigurable robotic system specifically designed to meet the demands of hybrid integration for miniaturized photonic and quantum System-in-Packages (SiPs). The proposed solution introduces a distinctive approach to ultra-high-resolution, multi-telerobotic assembly and inspection. By integrating key Industry 5.0 principles, it establishes a human-centric control framework that minimizes both physical and cognitive stress while ensuring the human operator remains in full control at all times.The robotic system features eight robots working simultaneously within a compact footprint of just 5×10 cm2. A comprehensive digital twin framework constitutes a central element of the robotic system. It encompasses the robotic workcell, the SiP under assembly, and the components to be integrated, ensuring precise adherence to design specifications. Key functionalities include automated path planning in a multi-robotic environment, collision avoidance in a densely packed workcell, and virtual fixtures to guide teleoperation, enhancing the operator’s control and interaction through advanced and intuitive human-machine interfaces (HMI). The proposed system meets the critical demands of ultra-high-resolution assembly for complex, high-value SiPs, providing high flexibility and ease of operation for small-batch manufacturing.

Learning from Demonstration (LfD) is a widely used approach for teaching robot motion, but more sophisticated strategies are required to address complex tasks such as surface processing. Sanding is an example where comprehensive strategies are necessary to ensure complete and efficient coverage of the surface of a workpiece. In this paper, we present a system that captures human motions and contact forces during surface processing using a powered sanding tool. We provide a publicly available dataset that consists of demonstrations for various geometric shapes with the goal to extract robot execution strategies through LfD from a variety of users. This is in contrast to conventional LfD, which generates a policy directly from one or multiple trajectories provided by a single user. Further, we provide a data analysis that reveals key insights into how humans adapt their strategies to different surface geometries and extract robot execution strategies from it. Finally, we conduct two basic robotic experiments justifying the approach of strategy extraction. Our findings contribute to the understanding of human surface-processing behavior and lay the foundation for developing more effective robotic surface processing strategies.

Surface finishing such as grinding or polishing is a time-consuming task, involves health risks for humans and is still largely performed by hand. Due to the high curvatures of complex geometries, different areas of the surface cannot be optimally reached by a simple strategy using a tool with a relatively large and flat finishing disk. In this letter, a planning method is presented that uses a variable contact point on the finishing disk as an additional degree of freedom. Different strategies for covering the workpiece surface are used to optimize the surface finishing process and ensure the coverage of concave areas. Therefore, an automatic segmentation method is developed to find areas with a uniform machining strategy based on the exact tool and workpiece geometry. Further, a method for planning coverage paths is presented, in which the contact area is modeled to realize an adaptive spacing between path lines. The approach was evaluated in simulation and practical experiments on the DLR SARA robot. The results show high coverage for complex freeform geometry and that adaptive spacing can optimize the overall process by reducing uncovered gaps and overlaps between coverage lines.

This study addresses a multi-objective optimization problem in the planning of uncertainty-aware sequence and motion for mechanical products with intricate structures and numerous contact areas. The proposed pipeline involves planning several elements, including assembly order of parts, object placement pose, grasp, and arm trajectory. To generate an optimized sequence and motion that satisfies multiple conditions under mandatory requirements, we use a multi-objective optimization algorithm inspired by Non-Dominated Sorting Genetic Algorithm III (NSGA-III), along with contact-rich robotic assembly-oriented constraints and objective functions. The proposed pipeline takes as input the CAD models of robot hardware, workspace, and assembled parts, conducts 3D geometrical and physical simulations of assembly motions, and then optimizes the assembly plan, including parts order, object placement pose, state transition, grasp, and trajectory for the real robot to execute. The key component of the proposed pipeline is the uncertainty-aware ConCERRT-based state transition planner. Our experiments on assembly planning for a chainsaw product demonstrated that the proposed method can generate constraint-satisfied assembly plans with a 100% success rate while lowering uncertainty in simulations.

This study addresses a multi-objective optimization problem in the planning of an uncertainty-aware sequence and motion for mechanical products with intricate structures and numerous contact areas. To generate an optimized sequence and motion that satisfies multiple conditions under mandatory requirements, we use a multi-objective optimization algorithm inspired by Non-Dominated Sorting Genetic Algorithm III, along with contact-rich robotic assembly-oriented constraints and objective functions. The proposed pipeline takes as input the CAD models of robot hardware, workspace, and assembled parts, conducts 3D geometrical and physical simulations of assembly motions, and then optimizes the assembly plan, including parts order, object placement pose, state transition, grasp, and trajectory for the real robot to execute. To obtain the uncertainty-aware sequence and motion, we incorporated a Contingent Contact-Exploring Rapidly-exploring Random Trees (ConCERRT)-based state transition planner and an objective function to evaluate the uncertainty in the multi-objective optimization algorithm. Our experiments on assembly planning for a chainsaw product demonstrated that the proposed method can generate constraint-satisfied assembly plans with a success rate of 99.2% while lowering the uncertainty in the simulations.

Smart manufacturing requires easily reconfigurable robotic systems to increase the flexibility in presence of market uncertainties by reducing the set-up times for new tasks. One enabler of fast reconfigurability is given by intuitive robot programming methods. On the one hand, offline skill-based programming (OSP) allows the definition of new tasks by sequencing pre-defined, parameterizable building blocks termed as skills in a graphical user interface. On the other hand, programming by demonstration (PbD) is a well known technique that uses kinesthetic teaching for intuitive robot programming, where this work presents an approach to automatically recognize skills from the human demonstration and parameterize them using the recorded data. The approach further unifies both programming modes of OSP and PbD with the help of an ontological knowledge base and empowers the end user to choose the preferred mode for each phase of the task. In the experiments, we evaluate two scenarios with different sequences of programming modes being selected by the user to define a task. In each scenario, skills are recognized by a data-driven classifier and automatically parameterized from the recorded data. The fully defined tasks consist of both manually added and automatically recognized skills and are executed in the context of a realistic industrial assembly environment.

Skill-based robotic systems can perform tasks more flexibly than typical industrial manipulators. These systems are equipped with a repertoire of reusable skills and take advantage of a knowledge base about their workspace. That being so, the robot can execute tasks composed of a combination of different skills, tools, and objects without having to be reprogrammed explicitly for each task. Despite its advantages, these systems are affected by modeling errors and an inaccurate knowledge base. Such issues lead to failures in production. Since automated error detection is still an open problem, they often have to be solved by a robot operator. That is generally done by accessing the implementation of the faulty task and determining what to change to achieve the desired outcome, which is time-consuming and requires expertise. The proposed work aims to provide the robot operator with a faster and more intuitive error recovery method for a skill-based system via GUI-assisted kinesthetic refinement of robot skills. Furthermore, partially automated error recovery strategies are included. First, the targeted skills can be composed of an arbitrary number of steps with corresponding reversion behaviors. Second, consecutive human corrections on different parts of a given object are analyzed to infer a possible object pose error. Experiments show that our method takes one-fourth of the time required for conventional manual correction.

This paper presents an automated framework for designing and reconfiguring robotic workcells in high-mix, lowvolume manufacturing scenarios. The framework utilizes a modular workcell with various tools and a flexible grid system, the Variable Workstation (VWS), and uses an ontological representation of available robotic systems and assembly tasks. The major contribution is a tool for automatic layout planning and reconfiguration, and its integration into the overall pipeline. The tool optimizes component placements for efficient task execution with minimal path lengths and maximized manipulability. Constraints guarantee the reachability of each task and absence of collisions. The framework is evaluated in a real scenario using tasks from the assembly of an electric chainsaw. Overall, the proposed framework provides an efficient and complete system of designing and reconfiguring robotic workcells for high-mix, low-volume manufacturing.

It lies in human nature to properly adjust the muscle force to perform a given task successfully. While transferring this control ability to robots has been a big concern among researchers, there is no attempt to make a robot learn how to control the impedance solely based on visual observations. Rather, the research on tele-impedance usually relies on special devices such as EMG sensors, which have less accessibility as well as less generalization ability compared to simple RGB webcams. In this letter, we propose a system for a vision-based tele-impedance control of robots, based on the approximately estimated muscle activation patterns. These patterns are obtained from the proposed deep learning-based model, which uses RGB images from an affordable commercial webcam as inputs. It is remarkable that our model does not require humans to apply any visible markers to their muscles. Experimental results show that our model enables a robot to mimic how humans adjust their muscle force to perform a given task successfully. Although our experiments are focused on tele-impedance control, our system can also provide a baseline for improvement of vision-based learning from demonstration, which would also incorporate the information of variable stiffness control for successful task execution.

Programming by Demonstration (PbD) is used to transfer a task from a human teacher to a robot, where it is of high interest to understand the underlying structure of what has been demonstrated. Such a demonstrated task can be represented as a sequence of so-called actions or skills. This work focuses on the recognition part of the task transfer. We propose a framework that recognizes skills online during a kinesthetic demonstration by means of position and force–torque (wrench) sensing. Therefore, our framework works independently of visual perception. The recognized skill sequence constitutes a task representation that lets the user intuitively understand what the robot has learned. The skill recognition algorithm combines symbolic skill segmentation, which makes use of pre- and post-conditions, and data-driven prediction, which uses support vector machines for skill classification. This combines the advantages of both techniques, which is inexpensive evaluation of symbols and usage of data-driven classification of complex observations. The framework is thus able to detect a larger variety of skills, such as manipulation and force-based skills that can be used in assembly tasks. The applicability of our framework is proven in a user study that achieves a 96% accuracy in the online skill recognition capabilities and highlights the benefits of the generated task representation in comparison to a baseline representation. The results show that the task load could be reduced, trust and explainability could be increased, and, that the users were able to debug the robot program using the generated task representation.

Compliant robot tasks such as grinding require a robot to use a specific control strategy and to consider a number of process parameters. It is demanding to program such behaviors from scratch. Therefore, so called contact skills can be employed that are pre-programmed control strategies, which are optimized for the intended task. With that level of abstraction, which is defining skills that are specific to the task, only the skill's parameters need to be identified and not the whole strategy to be implemented. In order to allow non-experts to transfer such complex behaviors to a robot, we present two different contact skills and how they are automatically parameterized by a human demonstration. This process learns the robot behavior in one shot while considering task goals, such as desired forces and motions. We evaluated our framework in the PyBullet physics simulator and showed that the parameterized skills follow the task goals while generalizing to changes in the environment.

The research community is puzzled with words like skill, action, atomic unit and others when describing robots’ capabilities. However, for giving the possibility to integrate capabilities in industrial scenarios, a standardization of these descriptions is necessary. This work uses a structured review approach to identify commonalities and differences in the research community of robots’ skill frameworks. Through this method, 210 papers were analyzed and three main results were obtained. First, the vast majority of authors agree on a taxonomy based on task, skill and primitive. Second, the most investigated robots’ capabilities are pick and place. Third, industrial oriented applications focus more on simple robots’ capabilities with fixed parameters while ensuring safety aspects. Therefore, this work emphasizes that a taxonomy based on task, skill and primitives should be used by future works to align with existing literature. Moreover, further research is needed in the industrial domain for parametric robots’ capabilities while ensuring safety.

Programming by demonstration is reaching industrial applications, which allows non-experts to teach new tasks without manual code writing. However, a certain level of complexity, such as online decision making or the definition of recovery behaviors, still requires experts that use conventional programming methods. Even though, experts cannot foresee all possible faults in a robotic application. To encounter this, we present a framework where user and robot collaboratively program a task that involves online decision making and recovery behaviors. Hereby, a task-graph is created that represents a production task and possible alternative behaviors. Nodes represent start, end or decision states and links define actions for execution. This graph can be incrementally extended by autonomous anomaly detection, which requests the user to add knowledge for a specific recovery action. Besides our proposed approach, we introduce two alternative approaches that manage recovery behavior programming and compare all approaches extensively in a user study involving 21 subjects. This study revealed the strength of our framework and analyzed how users act to add knowledge to the robot. Our findings proclaim to use a framework with a task-graph based knowledge representation and autonomous anomaly detection not only for initiating recovery actions but particularly to transfer those to a robot.

Technology has sufficiently matured to enable, in principle, flexible and autonomous robotic assembly systems. However, in practice, it requires making all the relevant (implicit) knowledge that system engineers and workers have – about products to be assembled, tasks to be performed, as well as robots and their skills – available to the system explicitly. Only then can the planning and execution components of a robotic assembly pipeline communicate with each other in the same language and solve tasks autonomously without human intervention. This is why we have developed the Factory of the Future (FoF) ontology. At its core, this ontology models the tasks that are necessary to assemble a product and the robotic skills that can be employed to complete said tasks. The FoF ontology is based on existing standards. We started with theoretical considerations and iteratively adapted it based on practical experience gained from incorporating more and more components required for automated planning and assembly. Furthermore, we propose tools to extend the ontology for specific scenarios with knowledge about parts, robots, tools, and skills from various sources. The resulting scenario ontology serves us as world model for the robotic systems and other components of the assembly process. A central runtime interface to this world model provides fast and easy access to the knowledge during execution. In this work, we also show the integration of a graphical user front-end, an assembly planner, a workspace reconfigurator, and more components of the assembly pipeline that all communicate with the help of the FoF ontology. Overall, our integration of the FoF ontology with the other components of a robotic assembly pipeline shows that using an ontology is a practical method to establish a common language and understanding between the involved components.

Learning from Demonstration (LfD) can significantly speed up the knowledge transfer from human to robot, which has been proven for relatively unconstrained actions such as pick and place. However, transferring contact or force-based skills (contact skills) to a robot is noticeably harder since force and position constraints need to be considered simultaneously. We propose a set of contact skills, which differ in the force and kinematic constraints. In a first user study, several subjects were asked to term a variety of force-based interactions, from which skill names were derived. In a second and third user study, the identified skill names are used to let a test group of subjects classify the shown interactions. To evaluate the skill recognition from the robot perspective, we propose a feature-based classification scheme to recognize such skills with a robotic system in a LfD setting. Our findings prove that humans are able to understand the meaning of the different skills and, using the classification pipeline, the robot is able to recognize the different skills from human demonstrations.

This work introduces a contact event pipeline to distinguish task-contact from Human-Robot interaction and collision during task execution. The increasing need for close proximity physical human-robot interaction (pHRI) in the private, health and industrial sector demands for new safety solutions. One of the most important issues regarding safe collaboration is the robust recognition and classification of contacts between human and robot. A solution is designed, that enables simple task teaching and accurate contact monitoring during task execution. Besides an external force and torque sensor, only proprioceptive data is used for the contact evaluation. An approach based on demonstrated task knowledge and the offset resulting from human interaction is designed to distinguish contact events from normal execution by a contact event detector. A contact type classifier implemented as Support Vector Machine is trained with the identified events. The system is set up to quickly identify contact incidents and enable appropriate robot reactions. An offline evaluation is conducted with data recorded from intended and unintended contacts as well as examples of task-contacts like object manipulation and environmental interactions. The system’s performance and its high responsiveness are evaluated in different experiments including a real world task.