This article presents a novel gaze-guided volitional control method for knee-ankle prostheses, designed to enhance the precision and intuitiveness of prosthetic control in complex locomotion tasks. The method utilizes an eye-tracking system coupled with an red, green, blue, depth (RGBD) camera to estimate 3-D gaze points, which are then used to construct a gaze attention map. This map serves as the basis for predicting human locomotion intentions, enabling the prosthesis to accurately step on the intended foot placement. Experimental validation was conducted with both a healthy subject and an amputee subject, focusing on level-ground walking and stair ascent tasks. The results demonstrated that the prosthesis could achieve precise foot placement, with a mean error of 1.22 +/- 3.14 cm for the healthy subject and 3.40 +/- 0.66 cm for the amputee subject during level-ground walking. In stair ascent tasks, the system successfully adapted to various gait patterns, with the prosthetic joint angles closely matching natural human movements. The proposed method also effectively reduced the root mean square error (RMSE) of 3-D gaze estimation to 2.70, 2.63, and 1.97 cm along the x, y, and z axes, respectively. These findings indicate that the gaze-guided volitional control method not only enhances the functionality of knee-ankle prostheses but also offers a more intuitive interface for users, aligning prosthetic actions with the user's visual focus and intentions.

The dynamic biomechanical characteristics of human walking joint torque make it highly challenging to develop lower limb exoskeletons capable of assisting multiterrain and multispeed walking. As a promising solution, joint torque-based assistance methods using deep learning models have been widely studied for their explainability and uniformity. However, the strong demand for diverse input data and inferior performance across different devices significantly constrain their real-world application. In this article, we propose future hip angle guided control (FHAGC)-a concise and explicit control framework for walking assistance of hip exoskeleton. The hip joint angle measured by motor encoder is first used to estimate future angle changes, which is then used to calculate the hip joint torque for subsequent walking assistance. Unlike end-to-end neural networks, the controller is composed of predictable, bounded functions with interpretable parameters, allowing users to have full control over the exoskeleton assistance. The FHAGC method achieved an average root mean square error (RMSE) of 0.185 Nm/kg across three subdatasets collected from different devices. Furthermore, compared to the Noexo condition, the proposed method reduced the net metabolic cost by 12.2% during treadmill walking and by 17.5% during real-world walking. The proposed framework provides a new perspective on exoskeleton control and holds great potential for real-world application.

Knowledge distillation (KD) aims to enhance the performance of lightweight student networks through the guidance of teacher models. However, the existing methods have deficiencies in two key aspects: First, these methods rely heavily on static representation alignment, failing to account for optimization sensitivity in different directions within the distillation subspace; second, they lack a fine-grained mechanism to align critical directional features. To address these issues, we propose Direction Sensitivity–based Knowledge Distillation method (DSKD), which can quantitatively measure the sensitivity of each direction to the loss function at different training stages and dynamically select the optimization direction accordingly. Meanwhile, we designed a directional sensitivities weighted distillation loss. By aligning the parameter matrices of the teacher and student models in the key directions, we can more effectively transfer knowledge and improve the distillation effect. We combined DSKD with multiple advanced distillation strategies and conducted an empirical evaluation in the GLUE benchmark and CIFAR-100. The results showed that this method could significantly improve the performance of existing distillation techniques.

Implantable brain-machine interfaces (iBMIs) have emerged as a groundbreaking neural technology for restoring motor function and enabling direct neural communication pathways. Despite their therapeutic potential in neurological rehabilitation, the critical challenge of neural decoder calibration persists, particularly in the context of transfer learning. Traditional calibration approaches assume the availability of extensive neural recordings, which is often impractical in clinical settings due to patient fatigue and neural signal variability. Furthermore, the inherent constraints of implanted neural processors-including limited computational capacity and power consumption requirements-demand streamlined processing algorithms. To address these clinical and technical challenges, we developed DMM-WcycleGAN (Dimensionality Reduction Model-Agnostic Meta-Learning based Wasserstein Cycle Generative Adversarial Networks), a novel neural decoding framework that integrates meta-learning principles with optimal transfer learning strategies. This innovative approach enables efficient decoder calibration using minimal neural data while implementing dimensionality reduction techniques to optimize computational efficiency in implanted devices. In vivo experiments with non-human primates demonstrated DMM-WcycleGAN's superior performance in mitigating neural signal distribution shifts between historical and current recordings, achieving a 3% enhancement in neural decoding accuracy using only ten calibration trials while reducing the calibration duration by over 70%, thus significantly improving the clinical viability of iBMI systems.

Predicting lower limb joint angles during human walking is crucial for enhancing the control performance of assistive wearable robots. Existing studies typically use surface electromyography for joint angle prediction. However, this sensor is easily affected by measurement conditions (such as skin environment and assembly position) and requires complex preprocessing of the measured signals. This paper proposes a method for predicting future hip joint angles using only motor encoders from exoskeleton robots. A joint angle predictor, trained on offline datasets, is introduced to capture comprehensive information through multi-scale and multi-span sampling of the joint angle time series. Moreover, to overcome issues such as user variability, sensor data drift, and terrain change in real-world exoskeleton applications, an adaptive strategy based on continual learning is employed to improve the prediction accuracy of JaP, referred to as AJaP. In offline joint angle prediction for time horizons of 50 ms, 100 ms, and 200 ms, the proposed JaP achieved mean absolute errors of [0.7846 +/- 0.0859 degrees, 1.5752 +/- 0.0666 degrees, 2.5887 +/- 0.0872 degrees], respectively. During exoskeleton-assisted walking experiments, AJaP achieved joint angle prediction mean absolute errors of [1.6132 +/- 0.2450 degrees, 2.1850 +/- 0.82190, 3.2091 +/- 1.1393 degrees] on level ground, ramp, and stair at 100 ms. With adaptive optimization, the prediction accuracy of AJaP improved by [59.76%, 60.21%, 64.77%] compared to directly deploying JaP

Human vision can instantaneously reflect motion intent during the execution of complex tasks. In rough terrain locomotion, intent prediction is achieved by foot placement prediction, which obtains intermediate gait parameters that indirectly reflect upcoming locomotion mode. This letter proposes a human vision aided framework to perform real-time foot placement prediction during dynamic walking on rough terrains, while precisely extracting step length, step height and foothold slope for the next step. In this letter, 3D human gaze is estimated by a depth-inertial eye tracking system. The Gaussian distribution-based gaze attention map is then used to model the temporal and spatial decay of human attention. To precisely measure gait parameters, a robust visual-inertial odometry algorithm is also employed to continuously estimate the camera pose. The real-time performance of the system is validated using data collected from online experiments, where five subjects are required to transverse rough terrains under different experimental setup. Experimental results demonstrate that the proposed approach achieves an average 97.4% intent prediction accuracy at 35% of the swing phase, with mean errors of 6.1 cm, 1.7 cm and $\boldsymbol{2.6<^>\circ }$ for estimating step length, step height and foothold slope, respectively. These findings indicate that the real-time system have the potential to further improve the stability and safety of the control of walking assistive robots.

In this work, we conducted research on deformable object manipulation by robots based on demonstration-enhanced reinforcement learning (RL). We present FADERL (FuzzyAugmented Demonstration-Embedded Reinforcement Learning),a novel framework for robotic manipulation of deformable objects that significantly improves reinforcement learning efficiency through synergistic unification of High-Dimensional Takagi-Sugeno-Kang (HTSK) fuzzy systems, Generative Adversarial Behavior Cloning (GABC), and Conditional Policy Learning (CPL). Compared to the Rainbow-DDPG baseline, FADERL achieves 2:01× higher global average reward and reduces standard deviation to 45% while requiring fewer computational resources. To address the high cost of human demonstration collection, we introduce a Nonlinear Model Predictive Control (NMPC)-based data augmentation method that generates high-quality demonstrations at minimal cost. Simulation results demonstrate that NMPC-generated demonstrations enable FADERL to achieve performance comparable to human demonstrations. Physical experiments on fabric manipulation tasks—diagonal folding, central-axis folding, and flattening—achieve success rates of 83.3%, 80.0%,and 96.7% respectively, validating our approach’s effectiveness in real-world scenarios. Unlike computationally intensive large-model approaches, FADERL provides a lightweight, task-specific solution with efficient adaptability, making it suitable for practical robotic applications in manufacturing.

Real-time sensing and processing of large-scale tactile information are crucial for enhancing the compliant interaction of embodied robots, especially in collaborative systems. However, existing robotic skin systems are limited by latency in high-throughput signal readout and intelligent reasoning, making robust real-time interaction challenging. Here, we present a flexible bimodal skin powered by edge intelligence, enabling real-time sensory perception, decision-making, and actuation based on large-area coverage. The modular bimodal skin integrates pressure and temperature sensors, providing full coverage on robotic arm with over 768 pressure and 75 temperature sensor units. A rapid, crosstalk-free readout interface is implemented using a frequency-encoding architecture. Furthermore, we develop a lightweight deep learning framework that enables real-time autonomous decision-making for the bimodal skin at the edge device. We demonstrate that our system facilitates smooth, adaptive interaction for individuals with mobility impairments, even under complex or emergency conditions. This technology opens a promising avenue for real-time perception and interaction in human-centered embodied robotics.

In human-robot systems, ensuring safety during force control in the presence of both internal and external disturbances is crucial. As a typical loosely coupled floating-base robot system, the supernumerary robotic leg (SRL) system is particularly susceptible to strong internal disturbances. To address the challenge posed by floating base, we investigated the dynamics model of the loosely coupled SRL and designed a hybrid position/force impedance controller to fit dynamic torque input. An efficient variable impedance control (VIC) method is developed to enhance human-robot interaction, particularly in scenarios involving external force disturbances. By dynamically adjusting impedance parameters, VIC improves the dynamic switching between rigidity and flexibility, so that it can adapt to unknown environmental disturbances in different states. An efficient real-time stability guaranteed impedance parameters generating network is specifically designed for the proposed SRL, to achieve shock mitigation and high rigidity supporting. Simulations and experiments validate the system's effectiveness, demonstrating its ability to maintain smooth signal transitions in flexible states while providing strong support forces in rigid states. This approach provides a practical solution for accommodating individual gait variations in interaction, and significantly advances the safety and adaptability of human-robot systems.

In cluttered environments, a human-following mobile robot must predict the motion intention of the followed human and take environmental obstacles into consideration. Consequently, it brings several challenges, such as the human’s detour direction prediction problem and the visibility maintenance problem for route planning. To overcome these problems, this paper proposes an integrated follow-ahead framework, in which the human’s detour behavior is predicted by the Leg Motion Model-based EKF (LMM-EKF) and the iterative human route search algorithm, followed by the Safe Corridor-based Model Predictive Controller (SCMPC) used to obtain the optimal control solution. Also, a new perspective about visibility is provided in this paper that, via placing multiple obstacle-free safe regions along the human’s intended direction without any complex preprocessing for the point cloud, SCMPC prevents the robot from collision and occlusion simultaneously based on the basic properties of the convex set. The validity of the proposed method is comprehensively verified through real-world experiments.

This paper presents a tactile-driven grip phase classification and adaptive haptic feedback system for soft robotic hand rehabilitation, addressing the limitations of conventional devices in dynamic contact state recognition and multimodal feedback. By integrating a hybrid LSTM-ResNet architecture with spatiotemporal feature fusion and Hidden Markov Model (HMM)-based state transition optimization, the Soft Grasp Classification Network (SGC Net) achieves real-time high-precision classification of grip phases (contact → slip → stable). A bio-inspired multimodal feedback module is designed, featuring linear vibration gradients (80-150 Hz) synchronized with object slip velocity and auditory state transition alerts, combined with visuo-tactile consistency stimulation to activate neuroplastic changes in motor-somatosensory cortical integration. Experimental results demonstrate robust performance across four typical grasping scenarios (sponge, cube, apple, bottle), providing a novel paradigm for intelligent rehabilitation devices.

This study proposes MFE-MTL, a multi-task learning framework based on low-channel surface electromyography (sEMG) signals for stroke rehabilitation. The method combines manual feature extraction with a joint classification-regression model to simultaneously perform hand gesture recognition and grasp force estimation. Compared with three widely used neural networks (FCN, CNN, and LSTM), MFE-MTL achieved the best performance under low-channel conditions, reaching a pearson correlation coefficient of 0.89 for force prediction and 95.41 % precision for gesture classification. The integrated multi-task design significantly improved recognition performance, demonstrating the effectiveness and practicality of MFE-MTL in resource-constrained rehabilitation scenarios.

As a massive amount of waste generated by global construction activities, robots capable of automatically recycling construction waste have become effective tools for conserving natural resources. However, the complex environments of construction sites and the high diversity of waste materials pose challenges for robotic inspection, target recognition, and grasping. Achieving precise waste recognition is a prerequisite for stable grasping operations. In construction sites, complicated lighting conditions and cluttered backgrounds result in low feature contrast of waste materials in traditional RGB images, severely hindering recognition accuracy. To eliminate illumination interference, this study employs a depth camera to acquire scene disparity information and develops a Quasi-Inverse Perspective Transformation (QIPT) module, which enhances geometric contrast between objects and background by generating transformed disparity (TD) maps. Experimental validation within a single-modal fusion framework based on UNet demonstrates that the TD modality significantly outperforms RGB and raw disparity modalities in segmentation performance, achieving an improvement of 12.8% in the mIoU metric.

A promising effective human-robot interaction in assistive robotic systems is gaze-based control. However, current gaze-based assistive systems mainly help users with basic grasping actions, offering limited support. Moreover, the restricted intent recognition capability constrains the assistive system's ability to provide diverse assistance functions. In this paper, we propose an open implicit intention recognition framework powered by Large Language Model (LLM) and Vision Foundation Model (VFM), which can process gaze input and recognize user intents that are not confined to predefined or specific scenarios. Furthermore, we implement a gaze-driven LLM-enhanced assistive robot system (MindEye-OmniAssist) that recognizes user's intentions through gaze and assists in completing task. To achieve this, the system utilizes open vocabulary object detector, intention recognition network and LLM to infer their full intentions. By integrating eye movement feedback and LLM, it generates action sequences to assist the user in completing tasks. Real-world experiments have been conducted for assistive tasks, and the system achieved an overall success rate of 41/55 across various undefined tasks. Preliminary results show that the proposed method holds the potential to provide a more user-friendly human-computer interaction interface and significantly enhance the versatility and effectiveness of assistive systems by supporting more complex and diverse task.

Spatiotemporal gait parameters play a crucial role in assessing the rehabilitation progress of patients with lower-limb impairments. However, existing gait parameter extraction methods heavily rely on wearable devices or high-precision motion capture systems, limiting their applicability in everyday environments. To overcome this limitation, we propose a novel gait parameter extraction system in which a RGB-D camera, mounted on a mobile robot, tracks human walking motion to capture lower-limb depth information. The system employs an RGB-D camera to capture lower-limb motion data and introduces the multiscale temporal attention enhancement network (MSTAE-Net) for gait parameter extraction. This model effectively mitigates the challenges of insufficient temporal feature representation and difficulty in capturing abrupt feature variations at gait phase transitions. Additionally, it segments gait cycles and extracts spatiotemporal gait parameters accordingly. Experiments were conducted on four healthy subjects and six patients, comparing the proposed method against four classical classification algorithms. The results demonstrate that our approach achieves an average gait phase classification accuracy of 96.56%, outperforming BiLSTM by 5.4%. For spatial gait parameters, the average error compared to footprint-based measurements is 0.02 m; for temporal gait parameters, the average error is 0.1 s compared to video frame-by-frame annotated gait events. The experimental results validate the proposed method's accuracy and robustness, demonstrating its feasibility and efficiency for nonwearable gait parameter extraction and its potential in clinical rehabilitation assessment and daily monitoring.

Current research on grasping state analysis in soft manipulators is limited and lacks broad applicability. In this article, we introduce a novel method that leverages multimodal data from flexible sensors and Inertial Measurement Units (IMUs) to develop a comprehensive grasping state analysis system for multidegree-of-freedom (multi-DOF) pneumatic soft manipulators. A Deep Spiking High-Dimensional Fuzzy Network (DSHTFN) algorithm is specifically designed to analyze the “3S” grasping states of soft manipulators—shaking, stable, and slipping—with greater depth and precision. A novel membership function, the BernoulliArctangent (B-Atan) function, has been designed to accommodate the unique characteristics of spiking input signals and support backpropagation capabilities. Experimental results demonstrate that our proposed method achieves accuracies of 95.66% and 96.05% in opposing-finger and three-fingered soft manipulator operations, respectively. Through comparative analysis with other algorithms, the superior performance of the B-Atan membership function and the DSHTFN approach in analyzing the grasping states of soft manipulators has been validated.

Hip joint moments during walking are the key foundation for hip exoskeleton assistance control. Most recent studies have shown estimating hip joint moments instantaneously offers a lot of advantages compared to generating assistive torque profiles based on gait estimation, such as simple sensor requirements and adaptability to variable walking speeds. However, existing joint moment estimation methods still suffer from a lack of personalization, leading to estimation accuracy degradation for new users. To address the challenges, this paper proposes a hip joint moment estimation method based on generalized moment features (GMF). A GMF generator is constructed to learn GMF of the joint moment which is invariant to individual variations while remaining decodable into joint moments through a dedicated decoder. Utilizing this well-featured representation, a GRU-based neural network is used to predict GMF with joint kinematics data, which can easily be acquired by hip exoskeleton encoders. The proposed estimation method achieves a root mean square error of 0.1180 +/- 0.0021 Nm/kg under 28 walking speed conditions on a treadmill dataset, improved by 6.5% compared to the model without body parameter fusion, and by 8.3% for the conventional fusion model with body parameter. Furthermore, the proposed method was employed on a hip exoskeleton with only encoder sensors and achieved an average 20.5% metabolic reduction (p < 0.01) for users compared to assist-off condition in level-ground walking. (c) 2025 The Author(s). Published by Elsevier B.V. on behalf of Shandong University. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Wearable devices are currently being used to reduce metabolism and assist people with disabilities for daily walking. For the elderly and disabled people, improving the walking stability of wearable devices is a crucial and unsolved research. In particular, the center of mass (CoM) trajectory can reflect the walking state as well as the stability of a person. For this reason, realizing the prediction of CoM trajectory under daily walking is promising to improve the wearable devices. In this article, a method for estimation and prediction of CoM during daily walking was proposed. A visual-inertial-odometry algorithm was used to obtain history CoM trajectories during walking, and the depth information from camera data was extracted by sequential distance embedding method and encode the information into terrain vectors. The trajectory vectors were patched with the corresponding terrain vectors and then realized the fusion of multi-modal data, which were then fed into a well-trained temporal convolution network to output the prediction results. With data from two outdoor datasets, the method in this paper was verified to be able to be used for CoM trajectory prediction for a variety of walking tasks and consume low computational cost. This method has the potential to be extensible in improving the stability of wearable devices, such as exoskeleton, powered prosthetic, and so on. Note to Practitioners-It is necessary to obtain the future state of human center of mass (CoM) to improve the stability of user with wearable devices. However, current methods for CoM trajectory prediction, especially with multi-modal data fusion are difficult to apply to this prediction task due to the high computational cost. This article addressed the problem of estimation and prediction of CoM under daily walking conditions and proposed a method to solve the above problems, which enable to deploy on wearable robots with limited computation resources. In our study, a visual-inertial-odometry algorithm was utilized to estimate the history CoM and ran on our edge device in 30 Hz. And a terrain feature embedding method was proposed, which was able to extract the terrain depth information from the camera and reduce the dimensionality of the camera data. With the help of the embedding technique, terrain feature could be efficiently fused with history CoM trajectories in a low-cost way and handle the terrain transition situation in the daily walking environment. Finally, a temporal convolution network was designed to generate the predicted results based on the fusion of history CoM trajectories and terrain vectors in average of 8ms. This is the first study to enable CoM trajectory prediction under multiple walking conditions using cameras and IMU sensors via deep learning. However, the VIO will be affected by the camera data quality under low-light conditions and our prediction only takes the height as the main factor affecting the CoM trajectory. We hope that this research will provide some insights to wearable devices. For example, it may serve as a high-level controller for exoskeleton or prostheses to generate natural motion by means of kinematic model.

Most existing gait parameter detection methods rely on wearable sensors, which impose an additional burden on patients and physicians. This study proposes a gait parameter detection robotic system based on lower limb depth information, utilizing an RGB-D camera to capture human motion data. To address gait phase classification in gait parameter detection, a Hierarchical Attention BiLSTM (HA-BiLSTM) model is proposed. By introducing a hierarchical attention mechanism, the model accurately captures fine-grained dynamic changes and global motion trends in gait sequences, thereby enhancing gait phase recognition capabilities. The gait cycle is then segmented based on gait phase results to extract spatiotemporal gait parameters. A 10-meter straight walking experiment was designed to validate the effectiveness of the proposed method. The results show that the gait phase recognition accuracy reached 96.26%, with an average error of 0.02 m for step length and step width, and an average error of 0.208 s for gait phase duration.

Autonomous obstacle avoidance in dynamic and unknown environments is essential for mobile robots, forming a prerequisite for many high-level tasks. However, existing methods are often constrained by the limited computational and energy resources of mobile platforms, making it challenging to ensure both obstacle avoidance performance and operational efficiency. In contrast, the human brain demonstrates remarkable adaptability and exploration capabilities in unfamiliar environments while consuming an extremely small amount of energy. Inspired by this, we propose a novel brain-inspired obstacle avoidance method that integrates brain-like perception, data fusion, and control strategies, offering both computational and energy efficiency. We employ a bio-inspired event camera to efficiently capture motion cues of objects in the environment and extract salient features using a spiking-driven self-attention-based event variational autoencoder (SDSA-EVAE). To compensate for the lack of depth information, we incorporate LiDAR to complement visual input through a multimodal fusion algorithm inspired by neuronal diversity. This functionality is achieved using a reinforcement learning framework that leverages a hybrid architecture of artificial neural networks (ANNs) and spiking neural networks (SNNs). Comparative simulations with baseline models demonstrate the effectiveness of our approach, while physical experiments validate its practical applicability. Furthermore, deployment on a neuromorphic processor confirms the energy efficiency of the method.