This paper presents a novel framework for Jacobian computation in motion optimization problems involving multi-link systems, where physical quantities are represented using higher-order time derivatives. In motion optimization of robots and humans, cost functions may incorporate higher-order time derivatives, such as jerk or the time variation of forces, to capture smoothness and perceptual characteristics, particularly in motion skill analysis and expressive behaviors, thereby necessitating Jacobian computations involving these quantities. However, such Jacobians are typically computed using numerical or automatic differentiation without explicitly exploiting the underlying multi-link structure, which can lead to increased computational cost and numerical instability. To address this limitation, we propose a structured Jacobian formulation for motion optimization, based on the comprehensive motion computation framework, in which physical quantities and their higher-order time derivatives are systematically represented along the multi-link structure. The proposed method systematically derives analytical expressions for Jacobians of kinematic and dynamic quantities, including momentum, forces, and joint torques, with respect to generalized coordinates and their higher-order derivatives. The resulting framework is applicable to both direct and inverse optimization. Through numerical experiments, we demonstrate that the proposed method improves computational efficiency compared to numerical and automatic differentiation, while achieving comparable accuracy. Furthermore, we demonstrate its effectiveness in inverse optimization by recovering cost function weights from motion data. Together, these results indicate that the proposed formulation provides a scalable and structured computational foundation for motion optimization involving higher-order time derivatives in multi-link systems.

Despite significant advances in bipedal locomotion, enabling humanoid robots to perform general whole-body tasks through meaningful interaction with their environments remains a challenging open problem. While deep reinforcement learning (RL) has recently demonstrated impressive results in dynamic walking — even on complex and unpredictable terrain — real-world utility demands that humanoids go beyond locomotion to execute task-oriented behaviors.In this work, we propose a framework for teaching humanoid robots to imitate humans doing useful tasks by training policies for tracking human motion references. Our approach leverages high-quality in-house motion capture (MoCap) data, from which we perform kinematic retargeting to project human trajectories onto a humanoid platform. Crucially, we adopt a hybrid learning paradigm: the policy is trained to track upper-body and root motions from the MoCap data, and receives additional supervision from a pre-trained omnidirectional walking expert. This expert guidance, implemented via a Behavior Cloning (BC) objective, ensures that leg motion respects dynamics and kinematic constraints of the humanoid. We train policies entirely in simulation and successfully transfer them to a real humanoid robot. We validate our method on a box loco-manipulation task, demonstrating effective sim-to-real transfer and marking a step toward more capable, task-driven humanoid behavior.

Robotic systems with redundant degrees of freedom can achieve the same task outcome using multiple configurations, resulting in solution sets that form manifolds in the configuration space. Existing approaches typically exploit such redundancy locally through Jacobian-based techniques to compute individual solutions or trajectories. While effective for solution computation, these methods do not retain a representation of the geometry of the solution set itself. In this work, we adopt a representation-centric approach to estimate the geometric structure of the solution space. We consider solution manifolds induced by general task-defining maps and construct an implicit scalar field over the configuration space, whose zero-level set corresponds to the solution manifold. To this end, we generate samples in the neighborhood of the solution manifold using a Jacobian-guided exploration strategy, which efficiently captures its local and global structure. The resulting implicit representation is defined over the configuration space and naturally induces a continuous, distance field that encodes proximity to the solution manifold. Experiments on a planar three-link robot and a seven-degree-of-freedom Franka manipulator demonstrate the effectiveness of the proposed representation. Furthermore, the framework enables consistent modeling of solution spaces across families of tasks with continuous variation.

Teleoperation enables human operators to control robots in remote environments, yet its integration with physical human–robot interaction (pHRI) for handling cumbersome objects remains limited. This work introduces a mixed reality (MR) teleoperation interface using an object-based control strategy, enabling the remote operator to manipulate a virtual point on the object rather than the robot’s tool center point. The approach was evaluated in collaborative manipulation of an object exceeding the robot’s payload capacity and compared with conventional control in a user study. Object-based control supported accurate, differentiated rotational behaviors and was rated more favorably in usability while maintaining low workload, indicating its potential for precise, intuitive manipulation of heavy or bulky objects.

In the research of robotic hands, various designs and controls have been explored from an engineering perspective to understand and emulate the functions of the human hand. However, approaches that utilize the functionality of the human hand as a standalone robotic avatar have rarely been considered. In this study, we propose a method for locomotion generation for a hand-shaped robotic avatar (HasRA), investigating the potential of the robot hand’s walking function by leveraging its degrees of freedom, which are typically used for object manipulation. We develop a robot hand and perform deep reinforcement learning to generate locomotion. The simulation results show that translational and rotational motions can be generated using reward functions derived from a small set of rules. Finally, we confirm that the translational motion can be reproduced on the real robot.

Manipulation with whole-body contact by humanoid robots offers distinct advantages, including enhanced stability and reduced load. On the other hand, we need to address challenges such as the increased computational cost of motion generation and the difficulty of measuring broad-area contact. We therefore have developed a humanoid control system that allows a humanoid robot equipped with tactile sensors on its upper body to learn a policy for whole-body manipulation through imitation learning based on human teleoperation data. This policy, named tactile-modality extended ACT (TACT), has a feature to take multiple sensor modalities as input, including joint position, vision, and tactile measurements. Furthermore, by integrating this policy with retargeting and locomotion control based on a biped model, we demonstrate that the life-size humanoid robot RHP7 Kaleido is capable of achieving whole-body contact manipulation while maintaining balance and walking. Through detailed experimental verification, we show that inputting both vision and tactile modalities into the policy contributes to improving the robustness of manipulation involving broad and delicate contact.

Robotic hands have been developed by many researchers to emulate human hand dexterity to enhance object manipulation. However, the mainstream of the robotic hand has been designed as part of the user’s body, and research scopes have been limited to improving functions to achieve human hands. In this paper, we propose Handoid, a novel robotic hand avatar that switches its morphology between acting as a part of a humanoid robot and an independent hand-shaped robot avatar physically separated from the main body. The prototype consists of a human-like robotic hand, an avatar robot acting as a user’s alter ego, a detachable mechanism for seamless attachment/detachment, and control software incorporating machine learning for generating walking motions. In the demonstration, users can experience the intuitive control of the telexistence robot, and the semi-autonomous walking of the robotic hand. This opens up novel possibilities for robotic hands.

Robot teleoperation (RTo) has emerged as a viable alternative to local control, particularly when human intervention is still necessary. This research aims to study the distance effect on user perception in RTo, exploring the potential of teleoperated robots for older adult care. We propose an evaluation of non-expert users' perception of long-distance RTo, examining how their perception changes before and after interaction, as well as comparing it to that of locally operated robots. We have designed a specific protocol consisting of multiple questionnaires, along with a dedicated software architecture using the Robotics Operating System (ROS) and Unity. The results revealed no statistically significant differences between the local and remote robot conditions, suggesting that robots may be a viable alternative to traditional local control.

Robot operation through body movements is intuitive, and upper limb motions are commonly used as input. However, when attempting to use body parts other than the upper limbs, such as the lower limbs, operability decreases due to differences in workspace. In this study, we propose a novel motion retargeting framework that enables the transfer of movements between limbs with different structures. Our method employs a mapping function based on the joint angles of limbs and performs optimization based on workspace analysis. We implemented lower-to-upper limb motion retargeting using reachability sphere maps analysis and validated the approach through task simulations. As a result, the proposed method allows the lower limb to perform tasks normally performed by the upper limb by adjusting the joint angle of the lower limb.

To enable humanoid robots to work robustly in confined environments, multi-contact motion that makes contacts not only at extremities, such as hands and feet, but also at intermediate areas of the limbs, such as knees and elbows, is essential. We develop a method to realize such whole-body multi-contact motion involving contacts at intermediate areas by a humanoid robot. Deformable sheet -shaped distributed tactile sensors are mounted on the surface of the robot's limbs to measure the contact force without significantly changing the robot body shape. The multi-contact motion controller developed earlier, which is dedicated to contact at extremities, is extended to handle contact at intermediate areas, and the robot motion is stabilized by feedback control using not only force/torque sensors but also distributed tactile sensors. Through verification on dynamics simulations, we show that the developed tactile feedback improves the stability of whole-body multi-contact motion against disturbances and environmental errors. Furthermore, the life-sized humanoid RHP Kaleido demonstrates whole-body multi-contact motions, such as stepping forward while supporting the body with forearm contact and balancing in a sitting posture with thigh contacts.

Anticipating a future scenario where the robot initiates its own actions and behaves voluntarily when collaborating with humans, our research focuses on human understanding and perception of unanticipated robot actions during physical human-robot interaction. While the current literature searches for key factors that make the human-robot collaboration successful, the question of how people experience the robot’s unanticipated action as cooperative or uncooperative seems to remain open. We designed a game-based experiment (N=35) where the participant played a “catch-falling-coins” game by moving a robotic arm. Our experiment introduced unanticipated robot actions in an “active session” where the robot targeted higher-valued coins without first informing the participants. Through semi-structured interviews and statistical analysis of questionnaires (Big Five Personality Test, SAM, NARS and CH33), we examined the participants’ understanding of the robot’s “intention” and their positive or negative perception of the robot as cooperative or uncooperative. Among the participants who understood that the robot’s “intention” was to catch the higher-valued coins, the majority of them reported a positive perception of the robot (cooperative or helpful) while this was not the case among those who did not understand the robot’s intention. We also observed relevant relationships between some personality traits and a person’s understanding of the robot’s intention. Qualitative analysis of the interviews allowed us to structure the process of perception change during the game into three phases: confusion, investigation, and adaptation. We believe that our research contributes to the study of human perception, and particularly to the relationship between a human’s understanding of unanticipated robot actions and their positive or negative perception of the robot.

This paper proposes a framework for measuring human motions involving surface contacts by collecting data from distributed tactile sensors and motion capture systems simultaneously. Although contacts play an important role in natural robot interaction with humans and environments, their high complexity makes contact-rich motions challenging for even advanced humanoid robots. One possible approach is to learn from humans who generate such motions with ease in their daily lives. While analysis of human contact motions can lead to understanding human motion strategy to improve robots’ motion capacity and robustness, access to human motion data including contacts is still limited. This paper addresses this issue by establishing a method for obtaining human motions with wide-area contacts. The contact information measured by the tactile sensors is mapped on the human body through position-orientation and force registration, and unified with synchronized body motion data. A series of experiments have been conducted to validate the physical quality of the force measurement and demonstrate that the proposed framework is effective in acquiring whole-body contact motions.

Deep reinforcement learning has seen successful implementations on humanoid robots to achieve dynamic walking. However, these implementations have been so far successful in simple environments void of obstacles. In this paper, we aim to achieve bipedal locomotion in an environment where obstacles are present using a policy-based reinforcement learning. By adding simple distance reward terms to a state of art reward function that can achieve basic bipedal locomotion, the trained policy succeeds in navigating the robot towards the desired destination without colliding with the obstacles along the way.

The advancement and development of human modeling have greatly benefited from principles used in robotics, for instance, multibody dynamics laid the foundations for physics engines of human movement simulation, and the robotics and control theory were used to contextualize human sensorimotor control. There are many common interests and interconnections between the fields of human modeling and robotics. In recent years, as robots have become safer and smarter, they actively participate in our lives and help us in various scenarios. Roboticists need tools and data from human modeling to build next-generation robots that better assist humans. In this survey, we focus on the connections between physical human-robot interaction and human modeling. On one hand, human neuromusculoskeletal and sensorimotor control models provide novel insights into the human response that robots can utilize to improve human performance. On the other hand, robots are becoming instrumental in quantifying the performance of the (neuro)musculoskeletal system. Thus, the combined use of human modeling and robotic methods in physical human-robot interaction can lead to both improved human understanding and functional assistance.