Abstract Physical human-robot interaction requires robots to respond to intent and support balance without explicit commands. In walking assistance, most systems rely on discrete interfaces or distal sensing, limiting their ability to engage continuous physical interaction and regulate whole-body dynamics. Here, we show that large-area tactile coupling at the human torso provides a physically grounded channel through which intention alignment and stability support are intrinsically unified. By enabling safe interaction at mechanically meaningful regions-waist and underarms-the robot directly participates in regulating coupled human-robot dynamics rather than reacting to detected events. Using a tactile-centered control framework on a walking support robot, we demonstrate that locomotion stability, reduced muscular effort, and balance loss mitigation emerge naturally from continuous tactile interaction, without explicit commands or independent safety mechanisms. These results suggest torso-level tactile coupling offers a principled pathway toward safer, more cooperative physical human-robot interaction.

Continuous in-hand manipulation is an important physical interaction skill, where tactile sensing provides indispensable contact information to enable dexterous manipulation of small objects. This work proposed a framework for end-to-end policy learning with tactile feedback and sim-to-real transfer, which achieved fine in-hand manipulation that controls the pose of a thin cylindrical object, such as a long stick, to track various continuous trajectories through multiple contacts of three fingertips of a dexterous robot hand with tactile sensor arrays. We estimated the central contact position between the stick and each fingertip from the high-dimensional tactile information and showed that the learned policies achieved effective manipulation performance with the processed tactile feedback. The policies were trained with deep reinforcement learning in simulation and successfully transferred to real-world experiments, using coordinated model calibration and domain randomization. We evaluated the effectiveness of tactile information via comparative studies and validated the sim-to-real performance through real-world experiments.

Large-scale tactile sensors facilitate mobile manipulators to perform close and safe physical human-robot interactions with the perception of physical contact. Particularly in applications such as elderly walking support, we introduce a dual-arm mobile manipulator equipped with tactile skin, enabling a more humanlike support mechanism. To make the robot support humans, we establish a task hierarchy; and to make it follow humans walking, we aim to strike a balance between tasks and the compliance to contact forces. If maintaining tasks with compliance in the null-space, the compliant motions will get constrained. In this article, we propose a method that shapes prioritized compliant behavior from null-space compliance (NS-C) to whole-body compliance for walking support. Our approach evaluates the feasibility of NS-C, and in barely feasible cases, hierarchically propagates infeasible force components into task space, allowing the robot to generate appropriate compliant motions. Using this method, we can shape various compliant behaviors through specific task settings. We validate the approach through experiments on the dual-arm mobile manipulator.

Robotic hand tool use has garnered significant attention from robotics researchers, because it enhances dexterity beyond the limitations imposed by manipulators with fixed tool configurations and human-involved manual tool changes. Despite extensive research, current methodologies predominantly focus on imitating human hand trajectories, often neglecting the pivotal role of tool-environment interaction. This study addresses this gap by exploring the task of cucumber peeling as a case study to implement contact-based demonstration strategies in robotic tool use. Our approach concentrates on the subtle tool contact behaviors that manifest through contact dynamics. Specifically, we select appropriate tool stiffness for the peeling tasks, which is captured via a handheld teaching device equipped with optical tactile sensors. Subsequently, object-level stiffness control strategies are employed to emulate these behaviors using a three-fingered robotic hand. Experimental results from real-world cucumber peeling trials substantiate our methodology, illustrating that the robotic hand can adjust contact through finger movements, thereby achieving humanlike peeling efficiency without necessitating alterations to the tool structure. This study not only demonstrates the feasibility of sophisticated tool use by robotic hands, but also highlights the critical importance of integrating tactile feedback to refine interaction with the environment.

Robots and prostheses are increasingly designed with curvilinear surfaces for functional, aesthetic, aerodynamic, and safety reasons. Electronic skins (e-skins) capable of sensing contact location and pressure across complex, non-developable surfaces are essential for empowering next-generation robots with tactile awareness. This will facilitate safe and natural human-machine interactions while enhancing object manipulation capabilities. Despite the evident advantages of conformal e-skins, current fabrication methods face significant challenges in realizing their full potential. In this paper, we introduce thermoforming as a technique to efficiently fabricate tactile sensitive e-skins that conform to curvilinear surfaces. The performance, repeatability and uniformity of the sensors are characterized in detail. We also present a custom calibration pipeline where accurate digital replicas of conformal e-skins are generated for use in simulations. Finally, we demonstrate the benefits of 3D e-skins in a tool manipulation task.

Anthropomorphic robotic arms, mimicking the structure and function of human arms, show great potential for helping people in various tedious and repetitive household tasks. However, such arms mostly consist of multiple serial links controlled independently by actuators at joints with high reduction ratios, posing challenges in household services in terms of load capacity, responsiveness, and safety. In this paper, we propose a high-performance anthropomorphic arm called TRX-Arm based on differential cable transmission, characterized by features of high dynamics, high load capacity, and inherent compliance. TRX-Arm is composed of three deferential cable-driven coupling joints and one independent roll joint. Thanks to the cable differential transmission, the joints are capable of achieving doubled torque and stiffness without replacing motors. To enhance safety in human-robot interaction, the actuators including motors, reducer, belt, and pulley are mounted at the shoulder near the base and drive the joints remotely using cables, thereby minimizing the inertia of the whole arm. The workspace of TRX-Arm has a volume of 1.56 m(3), much larger than that of the human arm. Real experiments show its capabilities including high repeatability and load capacity as well as high dynamic behavior of a dual-arm robot platform built with TRX-Arms.

This paper proposes a new approach for improving robotic manipulation tasks that require both precision and high compliance through multisensory fusion. By integrating visual, force, and tactile feedback, our approach enhances performance in delicate insertion tasks. We introduce a unified framework that combines information from these sensors to guide the entire manipulation strategies. Experiments in simulated and physical environments demonstrate that our method outperforms traditional single and dual-modality approaches regarding precision, gentle interactions, and robustness. We also provide a detailed analysis of the results to examine the role of each modality during manipulation. The experiment videos are available at https://sites.google.com/view/vft-fusion-insertion .

Tactile servoing is an effective approach to enabling robots to safely interact with unknown environments. One of the core problems in tactile servoing is to robustly converge the contact features to the desired ones via a dedicated controller. This paper proposes a Data-Driven Model Predictive Controller (DDMPC) to compute the motion command given the previous interaction experience and feature deviations in tactile space. Compared with the manually designed PID-based controller, the proposed controller depends on the sound control theory and its convergence is guaranteed from a computational perspective. It is applied to the balancing control of a rolling bottle on a robotic forearm covered by a custom tactile sensor array. The real experiment demonstrates the superior robustness of the proposed approach and shows its great potential for other tactile servoing scenarios with measurement noise, which is inevitable for current tactile sensors.

This article proposes a control framework for robots to apply their entire body as potential effectors in physical human-robot interaction (pHRI) tasks. This framework is implemented based on high-resolution electronic skin that covers the entire body of the robot. During pHRI, robots must respond appropriately to human intentions, necessitating advanced sensing capabilities to interpret tactile information effectively. However, both discerning human intention from such large-scale tactile data and accommodating interaction across the entire body's surface present challenges. In this article, we propose a method to convert the large-area contact on a link into a contact center and estimate the corresponding wrench. Furthermore, we assign soft priorities and desired trajectories to each contact point and solve for the optimal joint velocities through quadratic programming (QP). By enabling a dual-arm mobile manipulator to dance the waltz with a human, our control framework has been validated for its effectiveness in handling multiple large-area contacts and time-varying pHRI tasks.

It is a challenging task to localize and track an in-hand object in robotic domain. Researchers were mainly using the vision as major modality for extracting object’s pose. The vision approaches are fragile when the object is occluded by the robotic arm and hand. To this end, we propose a tactile-based DTI-Tracker (tracking object’s pose via Dynamic Tactile Interaction) approach and formalize the object’s tracking as a filter problem. An Extended Kalman Filter (EKF) is used to estimate the in-hand object pose exploiting the high spatial resolution tactile feedback. Given the initial estimation error, the proposed approach rapidly converges the estimation result to the real pose and the statistic evaluation shows the robustness of the proposed approach. We evaluate this method in physics simulation and real multi-fingered grasping setup while the object is static and movable. The proposed method is a potential tool to foster future research on dexterous manipulation using multifingered robotic hand.

Objects of daily life are designed to suit the human hand. Without major modifications to these objects and our environments, robots will need end-effectors with human hand-like configuration and dexterity to efficiently operate on them. Tight integration of tactile and proprioceptive sensors are also critical to ensure robust execution of manipulation policies without sacrificing range-of-motion. Reliability is also key, and a mechanically robust, easy to repair end-effector is important to minimize downtime. To meet these challenges, we designed a 13 degree-of-freedom anthropomorphic hand with over 1000 tactile sensing elements, named TRX-Hand5. Also embedded within are positional encoders and cable tension sensors to provide proprioceptive perception. TRX-Hand5 has a novel biomimetic topology with six small posture motors in the palm to replicate the function of intrinsic hand muscles and five large power motors in the forearm to play the role of forearm flexor muscles. The whole hand weighs 2.6 kg with its dimensions comparable to those of an adult male's hand and is capable of actuating its fingertips at over 200 degrees /s while exerting up to 22N of force. The system can be disassembled in modules for easy maintenance.

Tactile sensors are believed to be essential in robotic manipulation, and prior works often rely on experts to reason the sensor feedback and design a controller. With the recent advancement in data-driven approaches, complicated manipulation can be realised, but an accurate and efficient tactile simulation is necessary for policy training. To this end, we present an approach to model a commonly used pressure sensor array in simulation and to train a tactile-based manipulation policy with sim-to-real transfer in mind. Each taxel in our model is represented as a mass-spring-damper system, in which the parameters are iteratively identified as plausible ranges. This allows a policy to be trained with domain randomisation which improves its robustness to different environments. Then, we introduce encoders to further align the critical tactile features in a latent space. Finally, our experiments answer questions on tactile-based manipulation, tactile modelling and sim-to-real performance.

In this paper, we present a new approach to estimate the pose of an object being manipulated by a multi-fingered robotic hand. The method utilizes advanced tactile sensors with high spatial resolution to optimize the estimation of the object's pose using an Extended Kalman Filter (EKF) based approach. We defined and derived the state and measurement equations, as well as evaluated the estimation accuracy in grasping tasks. The approach is able to effectively account for the pose transition caused by tactile pushing, and the mapping from the object's pose to the contact position and normal direction as measured by the tactile sensor. The method was evaluated in multiple grasping experiments in simulation scenarios. Results show that the estimation can converge towards the ground truth in a relatively short period of time, with displacement and rotation errors remaining within acceptable levels. This new method has the potential to improve the accuracy and reliability of robotic grasping and manipulation tasks.

In this letter, we propose a novel framework for tactile-based dexterous manipulation learning with a blind anthropomorphic robotic hand, i.e. without visual sensing. First, object-related states were extracted from the raw tactile signals by a graph-based perception model - TacGNN. The resulting tactile features were then utilized in the policy learning of an in-hand manipulation task in the second stage. This method was examined by a Baoding ball task - simultaneously manipulating two spheres around each other by 180 degrees in hand. We conducted experiments on object states prediction and in-hand manipulation using a reinforcement learning algorithm (PPO). Results show that TacGNN is effective in predicting object-related states during manipulation by decreasing the RMSE of prediction to 0.096 cm comparing to other methods, such as MLP, CNN, and GCN. Finally, the robot hand could finish an in-hand manipulation task solely relying on the robotic own perception - tactile sensing and proprioception. In addition, our methods are tested on three tasks with different difficulty levels and transferred to the real robot without further training.

Advancement in human-robot interaction (HRI) is essential for the development of intelligent robots, but there lack paradigms to integrate remote control and tactile sensing for an ideal HRI. In this study, inspired by the platypus beak sense, we propose a bionic electro-mechanosensory finger (EM-Finger) synergizing triboelectric and visuotactile sensing for remote control and tactile perception. A triboelectric sensor array made of a patterned liquid-metal-polymer conductive (LMPC) layer encodes both touchless and tactile interactions with external objects into voltage signals in the air, and responds to electrical stimuli underwater for amphibious wireless communication. Besides, a three-dimensional finger-shaped visuotactile sensing system with the same LMPC layer as a reflector measures contact-induced deformation through marker detection and tracking methods. A bioinspired bimodal deep learning algorithm implements data fusion of triboelectric and visuotactile signals and achieves the classification of 18 common material types under varying contact forces with an accuracy of 94.4 %. The amphibious wireless communication capability of the triboelectric sensor array enables touchless HRI in the air and underwater, even in the presence of obstacles, while the whole system realizes high resolution tactile sensing. By naturally integrating remote contorl and tactile sensing, the proposed EM-Finger could pave the way for enhanced HRI in machine intelligence.

In conjunction with huge recent progress in camera and computer vision technology, camera-based sensors have increasingly shown considerable promise in relation to tactile sensing. In comparison to competing technologies (be they resistive, capacitive or magnetic based), they offer super-high-resolution, while suffering from fewer wiring problems. The human tactile system is composed of various types of mechanoreceptors, each able to perceive and process distinct information such as force, pressure, texture, etc. Camera-based tactile sensors such as GelSight mainly focus on high-resolution geometric sensing on a flat surface, and their force measurement capabilities are limited by the hysteresis and non-linearity of the silicone material. In this paper, we present a miniaturised dome-shaped camera-based tactile sensor that allows accurate force and tactile sensing in a single coherent system. The key novelty of the sensor design is as follows. First, we demonstrate how to build a smooth silicone hemispheric sensing medium with uniform markers on its curved surface. Second, we enhance the illumination of the rounded silicone with diffused LEDs. Third, we construct a force-sensitive mechanical structure in a compact form factor with usage of springs to accurately perceive forces. Our multi-modal sensor is able to acquire tactile information from multi-axis forces, local force distribution, and contact geometry, all in real-time. We apply an end-to-end deep learning method to process all the information.

Sensor matrices are essential in various fields including robotics, aviation, health care, and industrial machinery. However, conventional sensor matrix systems often face challenges such as limited reconfigurability, complex wiring, and poor robustness. To address these issues, we introduce a one-wire reconfigurable sensor matrix that is capable of conforming to three-dimensional curved surfaces and resistant to cross-talk and fractures. Our frequency-located technology, inspired by the auditory tonotopy, reduces the number of output wires from row × column to a single wire by superimposing the signals of all sensor units with unique frequency identities. The sensor units are connected through a shared redundant network, giving great freedom for reconfiguration and facilitating quick repairs. The one-wire frequency-located technology is demonstrated in two applications-a pressure sensor matrix and a pressure-temperature multimodal sensor matrix. In addition, we also show its potential in monitoring strain distribution in an airplane wing, emphasizing its advantages in simplified wiring and improved robustness.

The human somatosensory system is capable of extracting features with millimeter-scale spatial resolution and submillisecond temporal precision. Current technologies that can render tactile stimuli with such high definition are neither portable nor easily accessible. Here, we present a wearable electrotactile rendering system that elicits tactile stimuli with both high spatial resolution (76 dots/cm(2)) and rapid refresh rates (4 kHz), because of a previously unexplored current-steering super-resolution stimulation technique. For user safety, we present a high-frequency modulation method to reduce the stimulation voltage to as low as 13 V. The utility of our high spatiotemporal tactile rendering system is highlighted in applications such as braille display, virtual reality shopping, and digital virtual experiences. Furthermore, we integrate our setup with tactile sensors to transmit fine tactile features through thick gloves used by firefighters, allowing tiny objects to be localized based on tactile sensing alone.

Grasping of objects using multi-fingered robotic hands often fails due to small uncertainties in the hand motion control and the object's pose estimation. To tackle this problem, we propose a grasping adjustment strategy based on tactile seroving. Our technique employs feedback from a sensorized multi-fingered robotic hand to collaboratively servo the fingers and palm to achieve the desired grasp. We demonstrate the performance of our method through simulation and physical experiments by having a robot grasp different objects under conditions of variable uncertainty. The results show that our approach achieved a higher success rate and tolerated greater uncertainty than an open-looped grasp.

Reinforcement Learning (RL) methods have been widely applied for robotic manipulations via sim-to-real transfer, typically with proprioceptive and visual information. However, the incorporation of tactile sensing into RL for contact-rich tasks lacks investigation. In this paper, we model a tactile sensor in simulation and study the effects of its feedback in RL-based robotic control via a zero-shot sim-to-real approach with domain randomization. We demonstrate that learning and controlling with feedback from tactile sensor arrays at the gripper, both in simulation and reality, can enhance grasping stability, which leads to a significant improvement in robotic manipulation performance for a door opening task. In real-world experiments, the door open angle was increased by 45% on average for transferred policies with tactile sensing over those without it.