This paper proposes an algorithm for detecting and estimating the pose of top objects in a complex environment where thin metal circular plates are randomly stacked. In complex environments where multiple instances of the same object are randomly stacked, the robot needs to detect and compare objects to identify the top ones for grasping. Our approach involves a combination of deep learning-based instance segmentation and an overlap handling algorithm for precise top object detection. Subsequently, leveraging three-dimensional geometric data, we estimate the object’s pose by determining its plane. To validate the proposed algorithm, we constructed two environments consisting of objects with different sizes and thicknesses. The first experiment quantitatively validated the object detection and overlap handling algorithm. The second experiment quantitatively compared different plane estimation algorithms. The third experiment quantitatively compared the pose of objects using the G-ICP (Generalized Iterative Closest Point) algorithm and the proposed algorithm against the ground truth pose. Additionally, we performed a qualitative comparison by visualizing the poses estimated by each algorithm in the images. In the experimental results, the overlap handling algorithm had an average success rate of 84.21%. Additionally, pose estimation using G-ICP before plane estimation frequently resulted in issues like drift in the center point and frequent misalignment with areas other than the object. On the other hand, pose estimation using G-ICP after plane estimation and the proposed algorithm yielded similar performance with average ADD-S values of 6mm or less. However, the pose estimated using the proposed algorithm resulted in a minimum 0.25x reduction in execution time compared to the G-ICP algorithm.

Model-mediated teleoperation (MMT) is intended to improve system stability and transparency in the presence of time delay between the haptic device and the robot. Previous experimental researches, however, report that systems mediated using linear models become unstable when contacting nonlinear objects. This paper analytically shows the effect of model mismatch on the stability of the system. The analysis shows that the mismatch can generate surplus energy in the haptic system when the stiffness coefficient of the environment increases with the deformed depth. The analysis is verified experimentally in virtual environment with second-order polynomial stiffness object, a linear object with varying stiffness, and a nonlinear viscoelastic object described by Hunt-Crossley model. Results of experiments show that the additionally generated energy destabilizes the system when there is time delay. It is observed that MMT mediated by a linear model becomes unstable even when the object also has linear stiffness if the stiffness changes even once. The experiments with Hunt-Crossley model shows that the damping element may not prevent the instability due to model mismatch.

Model-mediated teleoperation (MMT) employs an environment model at the master side to compute feedback output to the master at a faster rate. This approach improves system stability in the presence of time delay. MMT, however, does not generally perform well if the employed model is not accurate. The model mismatch is unavoidable when the environment is unknown in advance or varies. This paper proposes MMT employing an adaptive model. The proposed method adaptively moves the reference point of the employed model, whereas the previous MMTs used reference points fixed to the surface of objects in the environment. This can make system stability independent of the time delay. Experiments show that the proposed method improves stability compared to the previous MMTs when there are model mismatches. User studies are conducted to compare the operator’s performance in two tasks, control of force exerted to objects in the environment, and discrimination of object stiffness. The result shows that the error in the forces applied to objects in the environment significantly decreases in the proposed method. Errors in forces rendered to the master are also improved by at least 20.2%. The experiment result also shows that subjects can discriminate up to 40.9% smaller differences in the stiffness than the previous MMT under the same time delay.

A haptic master with a 3-axis gimbal structure has been developed to control active-steering catheter robots. The operator's forearm may collide with a link of the master device during the operations. This paper reports a method to avoid the collision. Distance sensors are attached to the expected collision points, and artificial potential fields are defined around the sensors. The closest link to the ground is controlled as if the potential fields apply a repulsive force. Other links except the avoidance-controlled link are also controlled simultaneously to maintain the orientation of the master handle. This scheme is experimentally verified.

Forces applied to an instrument tip of a surgical robot can be estimated by measuring the reaction forces at the drape plate and the trocar. Friction, however, occurs at the rubber packing inside the trocar. The friction deteriorates estimation of the force applied to the instrument tip in the axial direction. This paper proposes a method to improve the accuracy of force estimation by designing a l-DOF force sensor at the drape plate and measuring the friction that occurs in the rubber packing. Rectangular parallelepiped connectors between the upper and lower sections of the drape plate are designed to concentrate the strain from the reaction force within the drape plate. The proposed drape plate sensor can measure the reaction force accurately regardless of friction due to surface contact with the upper slide because the reaction force is measured within the internal structure of the drape plate. The sensor measures the reaction force in the axial direction with an error of 3.28 %. It is possible to estimate the axial force applied to the instrument tip with error of 8.26 % by measuring both the reaction force at the drape plate sensor and the friction within the rubber packing.

Estimation of the force applied to the tip of a surgical instrument is necessary to render high-fidelity haptic feedback. Accuracy of the force estimation suffers from friction caused by the rubber packing inside the trocar. This paper proposes a 3-DOF force sensor installed to the trocar support. It is possible to estimate the 2-DOF radial force applied to the instrument tip and the 1-DOF axial friction occurring in the trocar by using the 3-DOF force sensor. Accuracy of the estimation, however, deteriorates due to the reaction moment that occurs at the trocar support. An I-shaped force sensor is designed to reduce coupling effect which is caused by the reaction moment. Design parameters of the sensor are optimized to minimize the coupling effect by using ANSYS software. The sensor is manufactured and calibrated using the least-square calibration method. L2 relative error of the estimation of the radial force applied to the instrument tip is less than 6.30 %. The axial reaction force corresponding to the friction is also estimated with relative error less than 8.63 %.

Force applied to the tip of a surgical instrument can be estimated by measuring reaction force to sensors attached to the drape plate and the trocar support. Effect of the gravity on the sensors attached to the instrument changes depending on the posture of the slave robot. It deteriorates the accuracy of the force estimation. This paper proposes a method to estimate the axial force applied to the instrument tip in consideration of the effect of the gravity. The effect of the gravity on the drape-plate sensor is analyzed. A calibration equation for the sensor is developed accounting the angle of pitch and yaw of the slave robot. It is possible to estimate the axial force with a L-2 relative error of less than 7.79 % regardless of the changes in the gravity effect to the instrument, which depends on the slave posture.

Haptic sensation delivered to doctor's hands during CT-guided needle intervention complements insufficient visual information. Exploitation of the haptic information is to be trained using simulation. A model is necessary to compute the reflective forces occurring when the needle tip contacts with layers of soft tissues such as skin, membrane, and other layers of tissues. This paper proposes a nonlinear viscoelastic model based on measurements using porcine soft tissues. A 6-axis force-torque sensor and a needle are attached to the end of a 6-DOF articulated robot to measure the layer forces. The measurement results show that the layer forces increase nonlinearly with displacement of the needle. The force is affected by change of the velocity. It is also shown that the reflective force is relaxed when the needle stops. A standard linear solid model which can describe the relaxation phenomena is modified to describe nonlinear damping and stiffness forces. Accuracy of the developed model is verified through comparison between the model and measured data using porcine tissues. The proposed model can describe the nonlinear stiffness and relaxation phenomena with relative error less than 3.8 %.

Strain gauges attached to the driving pulleys of the instrument of surgical robots allow estimation of the torque between the instrument and environment. Friction in the torque transmission, however, deteriorates accuracy of the estimation. This paper proposes a method to reduce the estimation error using a friction model and Butterworth low-pass filter. The friction model is developed based on Dahl model and reflects the characteristics of the particular driving mechanism. Experimental results show that the relative error can be reduced to 3.49

This paper proposes a method to estimate vertical interaction force to the end of the surgical instrument by measuring reaction force at the part supporting the trocar. Relation between the force to the trocar and the interaction force is derived using the beam theory. The vertical interaction force is modeled as a function of the reaction force to the trocar and the distance between the drape plate and the trocar. Experimental results show that error is induced by the asymmetric shape of the trocar tip because contact position between the instrument and the trocar tip is changed depending on the direction of the interaction force. The theoretical relation, therefore, is compensated and reduced. Average $L_2$ relative error of the estimated force in the x-direction and the y-direction is 5.81 % and 5.99 %, respectively.

Experienced doctors complement insufficient visual information during CT-guided needle intervention by using haptic sensation delivered to their hands. Training simulation with haptic feedback can enhance effect of the training. A model is necessary in such simulation to compute the friction force between the inserted needle and surrounding tissue in real-time. This paper proposes a mathematical model of the friction force based on the experiment results using porcine soft tissue. The friction force is measured by inserting a needle into the tissue using a 6-DOF articulated robot. The results show that the friction force increases gradually up to kinetic friction due to deformation of surrounding tissue. This is in contrast with friction force between rigid objects where kinetic friction appears immediately after movement. Deformation of the tissue also results that the friction has hysteresis over the velocity of the needle. Magnitude of the friction varies according to the moving direction of the needle. Dahl friction model is adopted and modified by adding a term for linear viscous friction based on the characteristics confirmed through experiments. Accuracy of the model is verified through comparison between the model and measured data from porcine tissue. The proposed model can describe friction force of various porcine tissue with root-mean-square error less than 0.13 N.

Gastrointestinal endoscopy simulations have been developed to train endoscopic procedures which require hundreds of practices to be competent in the skills. Even though realistic haptic feedback is important to provide realistic sensation to the user, most of previous simulations including commercialized simulation have mainly focused on providing realistic visual feedback. In this paper, we propose a novel design of portable haptic interface, which provides 2DOF force feedback, for the gastrointestinal endoscopy simulation. The haptic interface consists of translational and rotational force feedback mechanism which are completely decoupled, and gripping mechanism for controlling connection between the endoscope and the force feedback mechanism.

Skilled doctors overcome lack of visual information during CT-guided needle intervention by estimating the position of the needle using haptic sensation. Simulation to train the skill using haptic sensation has been developed, and realistic haptic sensation should be rendered in real-time to enhance the effect of training simulation. In order that, a model to calculate the reflective force and moment is necessary. This paper focuses on modeling the reflective bending moment that occurs when the angle of the inserted needle changes. Experiments to measure the reflective bending moment are performed by inserting a needle into porcine tissues using a 6-DOF articulated robot. As a result, it was confirmed that the reflective bending moment is linearly proportional to the bending angle and the effect of angular velocity change is negligible. On the basis of the measurement result, the reflective bending moment is modeled as a function of insertion length, bending angle of the needle, and position that the human force is applied. The accuracy is verified by comparing the model with measurement data using porcine tissues. The proposed model can describe the reflective bending moment with less than 5.4% error for single-layer tissue and 12.5% error for multi-layer tissue. The proposed model can be applied for real-time haptic rendering of training simulation of needle intervention.