Shape control of deformable objects is a challenging and important robotic problem. This article proposes a model-free controller using novel 3-D global deformation features based on modal analysis. Unlike most existing controllers using geometric features, our controller employs physically based deformation features designed by decoupling global deformation into low-frequency modes. Although modal analysis is widely adopted in computer vision and simulation, its usage in robotic deformation control is still an open topic. We develop a new model-free framework for the modal-based deformation control. Physical interpretation of the modes enables us to formulate an analytical deformation Jacobian matrix mapping the robot manipulation onto changes of the modal features. In the Jacobian matrix, unknown geometric and physical models of the object are treated as low-dimensional modal parameters, which can be used to linearly parameterize the closed-loop system. Thus, an adaptive controller with proven stability can be designed to deform the object while online estimating the modal parameters. Simulations and experiments are conducted using linear, planar, and volumetric objects under different settings. The results not only confirm the superior performance of our controller, but also demonstrate its advantages over the baseline method.

Automated laparoscope field of view (FoV) control in minimal invasive surgery (MIS) poses challenges, as existing solutions failed to address dynamic surgical FoV requirements across different phases and they neglected the misorientation effect or potential obstacles during the control process which raised safety concerns. In this letter, we propose a Gaussian mixture model (GMM)-based heuristic decision framework that can achieve safe automatic laparoscope control to provide the phase-specific requirements for surgeons. Leveraging GMM to fit the domain knowledge of instrument distributions, we formulate a general nonlinear constraint optimal control model to optimize the laparoscope motion. To overcome the time-consuming issue of raw nonlinear optimization solvers, we first employ a decoupling approach for misorientation control under remote center of motion (RCM) constraints and propose a novel heuristic decision algorithm for predicting the optimal target and the future trajectory in real-time while ensuring effective collision avoidance. Furthermore, a unified primitive motion controller is developed accordingly for the FoV control which is applicable to mechanical/soft-programmed RCM surgical robotic systems. Extensive validations on the surgical dVRK platform and the general UR5 robotic laparoscope system demonstrate the feasibility, versatility, and superiority of our framework for safe automated laparoscope control, providing personalized views based on the surgeon's recorded clinical videos.

Manipulation in tight environment is challenging but increasingly common in vision-guided robotic applications. The significantly reduced amount of available feedback (limited visual cues, field of view, robot motion space, etc.) hinders solving the hand-eye relationship accurately. In this article, we propose a new generic approach for online robot–camera calibration that could deal with the least feedback input available in tight environment: an arbitrarily restricted motion space and a single feature point with unknown position for the robot end-effector. We introduce the interactive perception to generate prescribed but tunable robot motions to reveal high-dimensional sensory feedback, which is not obtainable from static images. We then define the interactive feature plane (IFP), whose spatial property corresponds to the robot-actuating trajectories. A depth-free adaptive controller is proposed based on image feedback, where the converged orientation of IFP directly harvests the data for solving the hand–eye relationship. Our algorithm requires neither external calibration sensors/objects nor large-scale data acquisition process. Simulations demonstrate the validity of our method to accurately calibrate different types of robot under various system set-ups. In experiments, we show good results of our algorithm in terms of accuracy and consistency under tight motion space compared to existing approaches using external objects and/or optimization.

In this article, we present a novel and generic data-driven method to servo-control the 3-D shape of continuum and soft robots based on proprioceptive sensing feedback. Developments of 3-D shape perception and control technologies are crucial for continuum and soft robots to perform tasks autonomously in surgical interventions. However, owing to the nonlinear properties of continuum robots, one main difficulty lies in the modeling of them, especially for soft robots with variable stiffness. To address this problem, we propose a versatile learning-based adaptive shape controller by leveraging proprioception of 3-D configuration from fiber Bragg grating (FBG) sensors, which can online estimate the unknown model of continuum robot against unexpected disturbances and exhibit an adaptive behavior to the unmodeled system without priori data exploration. Based on a new composite adaptation algorithm, the asymptotic convergences of the closed-loop system with learning parameters have been proven by Lyapunov theory. To validate the proposed method, we present a comprehensive experimental study using two continuum and soft robots both integrated with multicore FBGs, including a robotic-assisted colonoscope and multisection extensible soft manipulators. The results demonstrate the feasibility, adaptability, and superiority of our controller in various unstructured environments, as well as phantom experiments.

Recently, fiber optic sensors such as fiber Bragg gratings (FBGs) have been widely investigated for shape reconstruction and force estimation of flexible surgical robots. However, most existing approaches need precise model parameters of FBGs inside the fiber and their alignments with the flexible robots for accurate sensing results. Another challenge lies in online acquiring external forces at arbitrary locations along the flexible robots, which is highly required when with large deflections in robotic surgery. In this paper, we propose a novel data-driven paradigm for simultaneous estimation of shape and force along highly deformable flexible robots by using sparse strain measurement from a single-core FBG fiber. A thin-walled soft sensing tube helically embedded with FBG sensors is designed for a robotic-assisted flexible ureteroscope with large deflection up to 270 degrees and a bend radius under 10 mm. We introduce and study three learning models by incorporating spatial strain encoders, and compare their performances in both free space and constrained environments with contact forces at different locations. The experimental results in terms of dynamic shape-force sensing accuracy demonstrate the effectiveness and superiority of the proposed methods.

Teleoperation control has been taken a pivotal role in robot-assisted surgical interventions. Unlike hand-eye coordination employed in the conventional laparoscopic systems, the eye-head coordination challenge arises with flexible robotic endoscopes, necessitating a distinct intuitive decision-making approach. In this paper, we introduce an intuitive teleoperation control scheme to address the eye-head coordination intricacies inherent in continuum endoscopic robots. The method achieves field of view (FOV) control consistent with surgeons' intuition for the robots based on the modelling of a virtual workspace (VW). By leveraging accurate shape feedback from embedded multi-core fiber Bragg gratings (FBGs), an observer-based anti-interference algorithm is implemented allowing a stable line of sight for robust motion control, even in challenging operational conditions with unknown disturbances. A class of robotic flexible endoscopes commonly used is analyzed specifically and deployed to validate the proposed approach. The efficacy and superiority of the proposed algorithm have been demonstrated through experimental results performed on a urology robotic system, encompassing trajectory tracking and disturbance resistance tasks.

Introduction The robotic-assisted surgical system has been widely used in hepatectomy. However, the effectiveness and feasibility of robotic-assisted hemi-hepatectomy (RH) has not been well-documented. Methods Patients who underwent RH or open hemi-hepatectomy (OH) performed by a single surgeon at our hospital between January 2010 and August 2023 were included in this study. A stabilized inverse probability of treatment weighting adjusted analysis was performed. Results Of the 163 consecutive patients identified, 60 underwent RH, and 103 underwent OH. After stabilized inverse probability of treatment weighting adjustment, RH demonstrated less blood loss than OH. In subgroup analyses, robotic-assisted left hemi-hepatectomy was associated with a shorter postoperative stay, a lower postoperative complication rate, and less blood loss compared with open left hemi-hepatectomy. While robotic-assisted right hemi-hepatectomy (RRH) was associated with less blood loss and a lower intraoperative blood transfusion rate, but a longer operation time compared with open right hemi-hepatectomy. Conclusions RH is a safe and effective technique. In addition to less blood loss, robotic-assisted left hemi-hepatectomy had advantages in postoperative complications and postoperative stay, while RRH had advantages in intraoperative blood transfusions. However, operation time was longer for RRH than for open right hemi-hepatectomy.

Objective: Although there have been studies conducted on the instantaneous remote center of motion (RCM) mechanism, the general closed-loop control method has not been studied. Thus, this article fills that gap and employs the advantages of this mechanism to develop a novel injection system. Methods: The injection prototype involves the instantaneous RCM mechanism, insertion unit and injection unit. The RCM system is investigated in the presence of time-varying axial stiffness of the screw drive and underactuated case. For safe interaction, compliance control is designed in the insertion system. The stability of all separate systems is investigated with the bounded parameter variation rate. The injection prototype and a robot end-effector were then combined to perform injection. Results: Our RCM prototype can achieve a large workspace, and its control effectiveness was verified by multiple frameworks and comparison with previous studies. Compliance-controlled insertion can achieve accurate depth regulation and zero-impedance control for manually operating the needle. With the help of three-dimensional reconstruction and hand/eye calibration, the manipulator can guide the injection prototype to a proper pose for injection of a face model. Conclusion: The injection prototype was successfully designed. The effectiveness of the whole control system was verified by simulations and experiments. The particular robotic injection task can be performed by the prototype. Significance: This article provides alternative schemes for developing an instantaneous RCM system, screw drive-based surgical tool, and robotic insertion with small needles.

Recent advancements toward perception and decision-making of flexible endoscopes have shown great potential in computer-aided surgical interventions. However, owing to modeling uncertainty and inter-patient anatomical variation in flexible endoscopy, the challenge remains for efficient and safe navigation in patient-specific scenarios. This paper presents a novel data-driven framework with self-contained visual-shape fusion for autonomous intelligent navigation of flexible endoscopes requiring no priori knowledge of system models and global environments. A learning-based adaptive visual servoing controller is proposed to online update the eye-in-hand vision-motor configuration and steer the endoscope, which is guided by monocular depth estimation via a vision transformer (ViT). To prevent unnecessary and excessive interactions with surrounding anatomy, an energy-motivated shape planning algorithm is introduced through entire endoscope 3-D proprioception from embedded fiber Bragg grating (FBG) sensors. Furthermore, a model predictive control (MPC) strategy is developed to minimize the elastic potential energy flow and simultaneously optimize the steering policy. Dedicated navigation experiments on a robotic-assisted flexible endoscope with an FBG fiber in several phantom environments demonstrate the effectiveness and adaptability of the proposed framework.

This paper proposes a novel graph-based framework for 3-D shape sensing of flexible medical instruments using multi-core fiber Bragg grating (FBG) sensors. Due to noisy signals, deformability of instruments, and environmental disturbances, conventional shape sensing methods using direct FBG measurements are far from accurate and stable, especially for long devices. The localization errors will substantially accumulate with the increase of sensing lengths. To tackle this challenge, we propose a generic 3-D shape graph to optimize the entire shape of flexible instruments globally and account for the accumulative errors in both spatial and temporal domains. By leveraging the geometry configurations of FBG cores as the measurement model, a robust dynamic filtering approach is introduced for iterative curvature and twist estimation, which guarantees edge constraints of the graph-based shape optimization. Dedicated experiments are processed to validate our sensing approach in both structured and unstructured environments, where a robotic-assisted colonoscope system embedded with a multi-core FBG fiber is manipulated for the evaluations of bending as well as paths following in 3-D space. The results demonstrate the superiority of our framework as a promising solution for 3-D shape reconstruction of flexible instruments and continuum robots in terms of accuracy, robustness, and fast response compared to state-of-the-art works.

Dexterous manipulation is important for modern logistics automation. Industrial robots remain inferior to human labor in this area, especially in handling unknown objects in unstructured scenarios. It is observed that the human wrist contributes significantly to the dexterous manipulation capability by agilely changing the hand orientation to approach the target object and adjusting wrist stiffness to adapt to dynamic interaction. Inspired by the human wrist, we propose a novel soft wrist to achieve hybrid motion/stiffness control in a compact and lightweight structure. The dexterous motion and stiffness adjustment are simultaneously enabled by a novel multicable jamming mechanism, which is achieved by a precalibration of the relationship between stiffness with robot length and bending angle to adapt to both compliant and forceful tasks. Dedicated experiments were performed to validate its motion dexterity and variable stiffness property. A soft grasping system was developed to demonstrate the manipulation capability of the soft wrist by tasking it with automatic packaging, which is challenging for robots. Overall, the proposed soft wrist demonstrates a promising solution to enhance robotic manipulation capability to a human comparable level.

Pneumatic soft robots are prized for their flexibility in achieving adaptable deformations and compliance adjustments. However, the conventional pumps used in these systems often rely on simplistic, sensorless, and on–off control mechanisms, which limit the potential of these robots. Drawing inspiration from the intricate functionality of the natural heart-pumping mechanism, we present an innovative and versatile pump that integrates an antagonistic pumping mechanism with a reinforcement-learning-powered control strategy. The antagonistic pump features dual chambers for expansion and deflation, valve-controlled interconnections, and distributed pressure sensors. This dedicated architecture enables a nuanced air exchange logic and pumping sequence, thereby facilitating a wide range of pneumatic actuation possibilities for pneumatic soft robots. Our pressure control method leverages the structural capabilities and degrees of freedom inherent in the pump, enhancing pumping efficiency and precision. Diverging from the traditional controllers, it autonomously evaluates the properties of unknown connected loads and dynamically adjusts pumping actions accordingly. Consequently, the pump provides multiple pumping modes, including load perception, rapid inflation, dual-load control, adaptive pumping across positive and negative pressure ranges, fine-tuning, and instantaneous pressure switching. Moreover, by iteratively executing the pumping cycle, the pump can extend its output pressure limits. We have successfully built and tested a prototype pump, validating its ability to achieve a broad range of pressures with precise and robust control. These results underscore the pump's potential to actuate diverse soft robots through multimode pumping, offering a pioneering solution for universal soft robot actuation and control.

Soft robots are gaining more and more attention owing to their inherent compliance and excellent flexibility, enabling new potentials for robots to address real-world chal-lenges in the manner of their natural counterparts. The soft con-tinuum robot is one typical example that reflects the successful inspiration achievement from biomimetics to robotics. However, due to the nonlinearity and ultra-high degrees of freedom of soft continuum robots, traditional kinematic modeling methods obtain unsatisfying precision, especially when robots interact with their surroundings. To precisely estimate the kinematic model of soft continuum robots, we propose a novel end-to-end proprioception method. Our method inputs time series data from the soft continuum robot actuators and outputs the end-tip position in three dimensions. The experimental results show that the proposed method can improve the kinematic accuracy compared to the constant curvature model-based method either with contact or in noncontact conditions.

Multi-link serial robots have gained growing popularity in robot-assisted surgeries with the advantages of flexibility and versatility. This type of robot has be introduced into the uterus manipulation in laparoscopic hysterectomy (UMLH) to mitigate the disadvantages of our previous robotic uterus manipulator. Compared with the other surgical tasks (e.g. laparoscope manipulation, orthopedic and neurosurgery surgeries), the UMLH has a more narrow workspace and a stringent task requirement. Selecting the proper initial configuration of the robot is a key issue for such kind of surgical operations. In this paper, we develop a scheme to select appropriate initial configurations for the UMLH. Moreover, our method can satisfy both the static RCM and dynamic RCM. In an dynamic RCM, the task path or trajectory depends on the robot initial pose and arbitrary (or indefinite) with respect to the robot base frame. We also proposed a criterion, i.e. configuration-type task capability (CTTC), to measure the task capability of a given configuration-type for the dynamic RCMs. Simulations are carried out to validate our scheme.

To realize a higher-level autonomy of surgical knot tying in minimally invasive surgery (MIS), automated suture grasping, which bridges the suture stitching and looping procedures, is an important yet challenging task needs to be achieved. This paper presents a holistic framework with image-guided and automation techniques to robotize this operation even under complex environments. The whole task is initialized by suture segmentation, in which we propose a novel semi-supervised learning architecture featured with a suture-aware loss to pertinently learn its slender information using both annotated and unannotated data. With successful segmentation in stereo-camera, we develop a Sampling-based Sliding Pairing (SSP) algorithm to online optimize the suture’s 3D shape. By jointly studying the robotic configuration and the suture’s spatial characteristics, a target function is introduced to find the optimal grasping pose of the surgical tool with Remote Center of Motion (RCM) constraints. To compensate for inherent errors and practical uncertainties, a unified grasping strategy with a novel vision-based mechanism is introduced to autonomously accomplish this grasping task. Our framework is extensively evaluated from learning-based segmentation, 3D reconstruction, and image-guided grasping on the da Vinci Research Kit (dVRK) platform, where we achieve high performances and successful rates in perceptions and robotic manipulations. These results prove the feasibility of our approach in automating the suture grasping task, and this work fills the gap between automated surgical stitching and looping, stepping towards a higher-level of task autonomy in surgical knot tying. Note to Practitioners—This paper aims to automate the suture grasping task in surgical knot tying by leveraging stereo visual guidance. To effectively robotize this procedure, it requires multidisciplinary knowledge to achieve suture segmentation, 3D shape reconstruction, and reliable automated grasping, while there are no existing works tackling this procedure especially using robots with RCM kinematics constraints and under complex environments. In this article, we propose a learning-driven method along with a 3D shape optimizer, which can conduct the suture segmentation and output its accurate spatial coordinates, serving as guidance for automated grasping operation. Apart from this, we introduce a unified function to optimize the grasping pose, and a vision-based grasping strategy is also proposed to intelligently complete this task. The experiments extensively validate the feasibility of our framework for automated suture grasp, and its successful completion can serve as a basis for the following looping manipulation, hence filling a step gap in robot-assisted knot tying. This framework can be also encapsulated into the medical robotic system, and by simply indicating (e.g. mouse click) the rough position of the suture’s tip in one camera frame, the overall framework can be initialized and further accomplish the suture grasping task, which further prompts a full autonomy of surgical knot tying in the near future.

In this paper, we propose a novel variable-length estimation approach for shape sensing of extensible soft robots utilizing fiber Bragg gratings (FBGs). Shape reconstruction from FBG sensors has been increasingly developed for soft robots, while the narrow stretching range of FBG fiber makes it difficult to acquire accurate sensing results for extensible robots. Towards this limitation, we newly introduce an FBG-based length sensor by leveraging a rigid curved channel, through which FBGs are allowed to slide within the robot following its body extension/compression, hence we can search and match the FBGs with specific constant curvature in the fiber to determine the effective length. From the fusion with the above measurements, a model-free filtering technique is accordingly presented for simultaneous calibration of a variable-length model and temporally continuous length estimation of the robot, enabling its accurate shape sensing using solely FBGs. The performances of the proposed method have been experimentally evaluated on an extensible soft robot equipped with an FBG fiber in both free and unstructured environments. The results concerning dynamic accuracy and robustness of length estimation and shape sensing demonstrate the effectiveness of our approach.

Robots have been used extensively in the battle against the COVID-19 pandemic since its outbreak. One prominent direction is the use of robots for swab sampling, which not only solves the shortage of medical staffs, but also prevents them from being infected during face-to-face sampling. However, a massive deployment of sampling robots is still not achievable due to their high costs, safety concerns, deployment complexity, and so on. In this letter, we propose a flexible, safe, and easy-to-deploy swab robot in a compact bench-top system. The robot can perform nasal/throat swab sampling tasks as dexterous as a human manual operation. The bio-mimetic rigid interior and soft exterior design guarantee the sampling robot with both flexibility and safety. Furthermore, the integration of 3-D fiber Bragg grating (FBG) based shape sensor and multi-axis force sensor may enhance the control performance. A dedicated constrained compliance control (CCC) algorithm was developed to tackle the unexpected interactions during sampling, which ensures the validity and safety of the sampling under disturbance. Various experiments are conducted to validate our system and prove its feasibility, flexibility, high safety, and efficiency for both nasal/throat swab sampling tasks. The proposed system is promising to be massive duplicated for robotic swab sampling.

In this paper, we present a novel and generic data-driven method to servo-control the 3-D shape of continuum and soft robots embedded with fiber Bragg grating (FBG) sensors. Developments of 3-D shape perception and control technologies are crucial for continuum robots to perform the tasks autonomously in surgical interventions. However, owing to the nonlinear properties of continuum robots, one main difficulty lies in the modeling of them, especially for soft robots with variable stiffness. To address this problem, we propose a versatile learning-based adaptive controller by leveraging FBG shape feedback that can online estimate the unknown model of continuum robot against unexpected disturbances and exhibit an adaptive behavior to the unmodeled system without priori data exploration. Based on a new composite adaptation algorithm, the asymptotic convergences of the closed-loop system with learning parameters have been proven by Lyapunov theory. To validate the proposed method, we present a comprehensive experimental study by using two continuum robots both integrated with multi-core FBGs, including a robotic-assisted colonoscope and multi-section extensible soft manipulators. The results demonstrate the feasibility, adaptability, and superiority of our controller in various unstructured environments as well as phantom experiments.

Laparoscopic hysterectomy is a common minimally invasive gynecologic surgery in which an assistant is required to manipulate the uterus during the procedure according to the verbal commands of the primary surgeon. However, uterus manipulation is a lengthy and laborious task, where human fatigue may lower operating safety. This paper presents a novel robot-enabled uterus manipulation system that provides accurate, tireless, and direct uterus manipulation. The system consists primarily of a 7-DoF robotic arm tailored for uterus manipulation in laparoscopic hysterectomy. The primary surgeon can directly control the robot using a footswitch pedal to move within the constraint of the Remote Center of Motion (RCM) via software and achieve the desired range of pitch and yaw motion. An additional rotational motion around the instrument axis is provided for twisting the uterus to enhance the exposure of the ligaments and peritoneum for resection. In addition, the robot arm consists of 7 compact modular actuators that provide the target payload for holding the uterus while maintaining a small footprint in a constrained operating environment. To facilitate operation, a passive mobile base is provided for supporting and positioning the robot. A 6-D force sensor is equipped that allows the assistant to quickly guide the robot into the work area and move the robot under RCM constraint using an admittance control approach. The uterus manipulation rod can be easily detached from the robot body by a quick pluggable mechanism for preoperative sterilization. A quick pluggable mechanism was proposed for easy separation and mounting of the uterus manipulation rod from the robot body for preoperative sterilization. Experiments were performed to measure the performance of the modular joints, RCM repeatability, and evaluate the feasibility of the robot in the simulated laparoscopic hysterectomy.

Sampling-based motion planning algorithms with task-space guidance have been proved to work efficiently in solving task-constrained manipulator planning problem as the planned end-effector path can satisfy the task constraints directly. However, planning in task-space rather than in configuration space (C-space) may lead to discontinuous motion in joint space, especially the connection of the bidirectional tree. In this paper, a Progressive Constraint Extension Bi-direction Rapidly Exploring Random Trees (PCE-BiRRT) algorithm is presented, which adopts a progressive constraint expansion strategy to make sure that the sampling configurations can satisfy the task constraint directly. For the smoothness of the bidirectional-tree connection, the proposed algorithm deals with the "connection failure problem" of the bidirectional exploring tree by limiting the connecting range and choosing a property redundancy resolution. The effectiveness of the PCE-BiRRT algorithm is tested on the robot operating system (ROS) platform by an orientation-constrained case.