In this paper, we evaluate the performance of our controller for flexible needle manipulation for percutaneous interventions in a finite element (FE) simulator. We investigate the use of electromagnetic (EM) tracking as needle tip pose feedback, and how artificial sensor noises can affect tracking performance of the controller. In our simulated study, the control system shows high targeting accuracy and robustness with an overall tip position error of 0.49 mm. The addition of needle tip orientation feedback further improves the targeting accuracy for deeper targets, with average error of 0.81 mm when only using position feedback, and 0.55 mm when using additional orientation feedback.

Cancer resection surgery is unsuccessful if tumor tissue is left behind in the surgical cavity. Identifying the residual cancer requires additional imaging or postoperative histological analysis. Photoacoustic imaging can be used to image both the surface and depths of the resection cavity; however, its performance hinges on consistent probe placement and stable acoustic and optical coupling. As intra-cavity deployment of photoacoustic imaging is largely uncharted, several potential embodiments warrant rigorous investigation. We address this need with an open-source robotic testbed for intraoperative tumor-bed inspection using photoacoustic imaging. The platform integrates the da Vinci Research Kit, depth imaging, and electromagnetic tracking to automate cavity scanning and maintain repeatable probe trajectories. Using tissue-mimicking phantoms, we (i) demonstrate a novel imaging embodiment for photoacoustic tumor-bed inspection and (ii) show how this testbed can be used to investigate and optimize tumor bed inspection strategies and configurations. This study establishes the feasibility of detecting and mapping residual cancer within a simulated surgical cavity. The primary contribution is the testbed itself, designed for integration with existing surgical navigation workflows and rapid prototyping. This testbed serves as an essential foundation for systematic evaluation of photoacoustic, robot-assisted strategies for improving intraoperative margin assessment.

This paper presents a flexible needle guidance system and its workflow that enables registration of computed tomography (CT) and electromagnetic (EM) tracking systems with a finite element (FE) simulator for needle-based percutaneous spinal injections. CT is used only pre- and postoperatively for surgical planning and confirmation, while EM tracking is combined intraoperatively with an FE-based needle controller to track the planned needle trajectory and avoid obstacles. Evaluation of the proposed system using a multi-layer soft tissue phantom shows an average targeting accuracy of 0.4 mm.

This paper presents a unified framework for autonomous flexible needle control in soft tissues using real-time finite element (FE) simulation and cross-entropy (CE) optimization. The method combines a sampling-based model predictive controller (MPC) for trajectory tracking with a kinematic-based bang-bang strategy to coordinate needle insertion, lateral adjustments, and bevel rotations. Sparse electromagnetic (EM) tracking feedback enables needle state reconstruction and compensates for model uncertainties. Experiments in plastisol and ex vivo chicken breast phantoms show sub-millimeter targeting accuracy, with respective targeting errors 0.16 ± 0.29 mm and 0.22 ± 0.78 mm as reported by the tracker.

Prostate cancer diagnosis continues to encounter challenges, often due to imprecise needle placement in standard biopsies. Several control strategies have been developed to compensate for needle tip prediction inaccuracies, however none were compared against each other, and it is unclear whether any of them can be safely and universally applied in clinical settings. This paper compares the performance of two resolved-rate controllers, derived from a mechanics-based and a data-driven approach, for bevel-tip needle control using needle shape manipulation through a template. We demonstrate for a simulated 12-core biopsy procedure under model parameter uncertainty that the mechanics-based controller can better reach desired targets when only the final goal configuration is presented even with uncertainty on model parameters estimation, and that providing a feasible needle path is crucial in ensuring safe surgical outcomes when either controller is used for needle shape manipulation.

We present a novel approach for minimally invasive flexible needle manipulations by pairing a real-time finite element simulator with the cross-entropy method. Additionally, we demonstrate how a kinematic-driven bang-bang controller can complement the control framework for better tracking performance. We show how electromagnetic (EM) tracking can be readily incorporated into the framework to provide controller feedback. Tissue phantom experiment with EM tracking shows the average targeting error is 0.16 ± 0.29mm.

Image-guided robotic interventions involve the use of medical imaging in tandem with robotics. SlicerROS2 is a software module that combines 3D Slicer and robot operating system (ROS) in pursuit of a standard integration approach for medical robotics research. The first release of SlicerROS2 demonstrated the feasibility of using the C++ API from 3D Slicer and ROS to load and visualize robots in real time. Since this initial release, we've rewritten and redesigned the module to offer greater modularity, access to low-level features, access to 3D Slicer's Python API, and better data transfer protocols. In this paper, we introduce this new design as well as four applications that leverage the core functionalities of SlicerROS2 in realistic image-guided robotics scenarios.

Head and neck cancers are the seventh most common cancers worldwide, with squamous cell carcinoma being the most prevalent histologic subtype. Surgical resection is a primary treatment modality for many patients with head and neck squamous cell carcinoma, and accurately identifying tumor boundaries and ensuring sufficient resection margins are critical for optimizing oncologic outcomes. This letter presents an innovative autonomous system for tumor resection (ASTR) and conducts a feasibility study by performing supervised autonomous midline partial glossectomy for pseudotumor with millimeter accuracy. The proposed ASTR system consists of a dual-camera vision system, an electrosurgical instrument, a newly developed vacuum grasping instrument, two 6-DOF manipulators, and a novel autonomous control system. The letter introduces an ontology-based research framework for creating and implementing a complex autonomous surgical workflow, using the glossectomy as a case study. Porcine tongue tissues are used in this study, and marked using color inks and near-infrared fluorescent (NIRF) markers to indicate the pseudotumor. ASTR actively monitors the NIRF markers and gathers spatial and color data from the samples, enabling planning and execution of robot trajectories in accordance with the proposed glossectomy workflow. The system successfully performs six consecutive supervised autonomous pseudotumor resections on porcine specimens. The average surface and depth resection errors measure $0.73\pm 0.60$ $\text{mm}$ and $1.89\pm 0.54$ $\text{mm}$ , respectively, with no positive tumor margins detected in any of the six resections. The resection accuracy is demonstrated to be on par with manual pseudotumor glossectomy performed by an experienced otolaryngologist.

Teleoperation of robots in space is challenging due to high latency and limited workspace visibility. Previously, the Interactive Planning and Supervised Execution (IPSE) and Augmented Virtuality systems were developed to reduce failure risk. These tools were visualized on a 3D da Vinci surgical console and operated using the da Vinci manipulators or visualized on conventional monitors and operated with a keyboard and mouse. Experimental studies indicated operator preference for the latter. In this work, we develop a 3D virtual reality (VR) interface for IPSE, implemented on a Meta Quest 2 head-mounted display (HMD), and evaluate it against the prior 2D, keyboard-and-mouse-based interface. The results demonstrate improved operator load with the 3D VR interface, with no decrease in task performance, while also providing cost and portability benefits compared to the conventional 2D interface.

Breast conserving surgery (BCS) is a surgical intervention for breast cancer where the surgeon resects the primary tumor and preserves the surrounding healthy tissue. These procedures have a high failure rate because it is often difficult to localize breast tumor boundaries intraoperatively. Emerging surgical robotic technology demonstrates promise for addressing such challenges. Hand-over-hand control is a design concept that involves cooperative handling of a surgical instrument by the surgeon and a robot. These systems can be used to reduce hand-tremor, improve dexterity, and implement virtual guidance [1]. One way to do this is with a virtual fixture (VF) which is a computer-generated constraint that is communicated from robotic devices to human operators via force and position signals [2]. A forbidden region VF can be used to enforce a safety region or “no-go zone” that can prevent the operator from damaging delicate or critical anatomy. In this paper, we use an Omni Bundle robot (formerly known as a Phantom Omni) to introduce haptic feedback in BCS. In doing so, we demonstrate a new and simplified approach to implementing a VF using open-source software tools such as 3D Slicer and robot operating system (ROS). We also make use of an existing surgical navigation platform that is used for BCS and describe how to deploy this system to encourage adaptation to other clinical applications.

Autonomous robotic surgery has the potential to provide efficacy, safety, and consistency independent of individual surgeon’s skill and experience. Autonomous anastomosis is a challenging soft-tissue surgery task because it requires intricate imaging, tissue tracking, and surgical planning techniques, as well as a precise execution via highly adaptable control strategies often in unstructured and deformable environments. In the laparoscopic setting, such surgeries are even more challenging because of the need for high maneuverability and repeatability under motion and vision constraints. Here we describe an enhanced autonomous strategy for laparoscopic soft tissue surgery and demonstrate robotic laparoscopic small bowel anastomosis in phantom and in vivo intestinal tissues. This enhanced autonomous strategy allows the operator to select among autonomously generated surgical plans and the robot executes a wide range of tasks independently. We then use our enhanced autonomous strategy to perform in vivo autonomous robotic laparoscopic surgery for intestinal anastomosis on porcine models over a 1-week survival period. We compared the anastomosis quality criteria—including needle placement corrections, suture spacing, suture bite size, completion time, lumen patency, and leak pressure—of the developed autonomous system, manual laparoscopic surgery, and robot-assisted surgery (RAS). Data from a phantom model indicate that our system outperforms expert surgeons’ manual technique and RAS technique in terms of consistency and accuracy. This was also replicated in the in vivo model. These results demonstrate that surgical robots exhibiting high levels of autonomy have the potential to improve consistency, patient outcomes, and access to a standard surgical technique.

Developing image-guided robotic systems requires access to flexible, open-source software. For image guidance, the open-source medical imaging platform 3D Slicer is one of the most adopted tools that can be used for research and prototyping. Similarly, for robotics, the open-source middleware suite robot operating system (ROS) is the standard development framework. In the past, there have been several "ad hoc" attempts made to bridge both tools; however, they are all reliant on middleware and custom interfaces. Additionally, none of these attempts have been successful in bridging access to the full suite of tools provided by ROS or 3D Slicer. Therefore, in this paper, we present the SlicerROS2 module, which was designed for the direct use of ROS2 packages and libraries within 3D Slicer. The module was developed to enable real-time visualization of robots, accommodate different robot configurations, and facilitate data transfer in both directions (between ROS and Slicer). We demonstrate the system on multiple robots with different configurations, evaluate the system performance and discuss an image-guided robotic intervention that can be prototyped with this module. This module can serve as a starting point for clinical system development that reduces the need for custom interfaces and time-intensive platform setup.

Malaria causes more than 200 million clinical illnesses and 45 million deaths every year, making mass production of an effective vaccine increasingly urgent. A Plasmodium falciparum sporozoites (PfSPZ) based vaccine has been proved to be a promising choice to defend against the malaria pandemic. However, large scale industrial production of PfSPZ vaccine is currently sub-optimal as the process for sporozoite extraction from salivary glands of infected mosquitoes is performed by manual microdissection, a relatively inefficient process requiring many hours of training. This paper reports continued progress in our development of a robotic system for automating the extraction of salivary glands. Compared to our previous versions, the structure of the new system is optimized to be more compact and allows different steps in workflow to be performed in parallel. We also report continued progress in developing key subsystem components. Finally, experiments show encouraging results with success rates of 90% in robotic mosquito manipulation, 95.6% in salivary gland extraction and 92.7% in debris disposal.

Malaria is a worldwide scourge, and the broad deployment of an effective vaccine would improve the lives of millions of people. A vaccine based on Plasmodium falciparum (PfSPZ) sporozoites extracted from the salivary glands of infected mosquitoes shows significant promise. However, the large-scale industrial production of PfSPZ-based vaccines will benefit from automation of the key step of extracting sporozoites from mosquito salivary glands that is currently performed by manual microdissection. In this work, we demonstrate a robotic system prototype for extracting salivary glands from mosquitoes to streamline vaccine production and reduce the need for operators. In the proposed system, mosquitoes are decapitated in an automated robotic pick-place-decapitate process, then a squeezer apparatus extracts mosquito salivary glands from the body. Mosquito detection and body part localization are performed by computer vision methods. The software allows system operation in simulation and on the robotic hardware, which facilitates subsystem development and integration. Experiments show encouraging results with success rates of 93% in robotic mosquito manipulation and 87.1% in salivary gland extraction. The system has the potential to improve the efficiency of PfSPZ vaccine production with significant gains in throughput and reduction in training times for a highly deskilled initial manual step. Further, this system is expected to pave the way for a more mature future system.

Approaches to robotic manufacturing, assembly, and servicing of in-space assets range from autonomous operation to direct teleoperation, with many forms of semi-autonomous teleoperation in between. Because most approaches require one or more human operators at some level, it is important to explore the control and visualization interfaces available to those operators, taking into account the challenges due to significant telemetry time delay. We consider one motivating application of remote teleoperation, which is ground-based control of a robot on-orbit for satellite servicing. This paper presents a model-based architecture that: 1) improves visualization and situation awareness, 2) enables more effective human/robot interaction and control, and 3) detects task failures based on anomalous sensor feedback. We illustrate elements of the architecture by drawing on 10 years of our research in this area. The paper further reports the results of several multi-user experiments to evaluate the model-based architecture, on ground-based test platforms, for satellite servicing tasks subject to round-trip communication latencies of several seconds. The most significant performance gains were obtained by enhancing the operators’ situation awareness via improved visualization and by enabling them to precisely specify intended motion. In contrast, changes to the control interface, including model-mediated control or an immersive 3D environment, often reduced the reported task load but did not significantly improve task performance. Considering the challenges of fully autonomous intervention, we expect that some form of teleoperation will continue to be necessary for robotic in-situ servicing, assembly, and manufacturing tasks for the foreseeable future. We propose that effective teleoperation can be enabled by modeling the remote environment, providing operators with a fused view of the real environment and virtual model, and incorporating interfaces and control strategies that enable interactive planning, precise operation, and prompt detection of errors.

Autonomous robotic surgery has the potential to provide efficacy, safety, and consistency independent of individual surgeons skill and experience. Autonomous soft-tissue surgery in unstructured and deformable environments is especially challenging as it necessitates intricate imaging, tissue tracking and surgical planning techniques, as well as a precise execution via highly adaptable control strategies. In the laparoscopic setting, soft-tissue surgery is even more challenging due to the need for high maneuverability and repeatability under motion and vision constraints. We demonstrate the first robotic laparoscopic soft tissue surgery with a level of autonomy of 3 out of 5, which allows the operator to select among autonomously generated surgical plans while the robot executes a wide range of tasks independently. We also demonstrate the first in vivo autonomous robotic laparoscopic surgery via intestinal anastomosis on porcine models. We compared the criteria including needle placement corrections, suture spacing, suture bite size, completion time, lumen patency, and leak pressure between the developed system, manual laparoscopic surgery, and robot-assisted surgery (RAS). The ex vivo results indicate that our system outperforms expert surgeons and RAS techniques in terms of consistency and accuracy, and it leads to a remarkable anastomosis quality in living pigs. These results demonstrate that surgical robots exhibiting high levels of autonomy have the potential to improve consistency, patient outcomes, and access to a standard surgical technique.

This paper presents a dual-arm suturing robot. We extend the Smart Tissue Autonomous Robot (STAR) with a second robot manipulator, whose purpose is to manage loose suture thread, a task that was previously executed by a human assistant. We also introduce novel near-infrared fluorescent (NIRF) sutures that are automatically segmented and delimit the boundaries of the suturing task. During ex-vivo experiments of porcine models, our results demonstrate that this new system is capable of outperforming human surgeons in all but one metric for the task of vaginal cuff closure (porcine model) and is more consistent in every aspect of the task. We also present results to demonstrate that the system can perform a vaginal cuff closure during an in-vivo experiment (porcine model).

INTRODUCTION AND OBJECTIVE:The inconsistency and difficulty of suturing in minimally invasive surgeries stem from the limitations in tool motion and visualization. In pelvic surgery, difficulty is ...

Background For autonomous robot-delivered surgeries to ever become a feasible option, we recommend the combination of human-centered artificial intelligence (AI) and transparent machine learning (ML), with integrated Gross anatomy models. This can be supplemented with medical imaging data of cadavers for performance evaluation. Methods We reviewed technological advances and state-of-the-art documented developments. We undertook a literature search on surgical robotics and skills, tracing agent studies, relevant frameworks, and standards for AI. This embraced transparency aspects of AI. Conclusion We recommend "a procedure/skill template" for teaching AI that can be used by a surgeon. Similar existing methodologies show that when such a metric-based approach is used for training surgeons, cardiologists, and anesthetists, it results in a >40% error reduction in objectively assessed intraoperative procedures. The integration of Explainable AI and ML, and novel tissue characterization sensorics to tele-operated robotic-assisted procedures with medical imaged cadavers, provides robotic guidance and refines tissue classifications at a molecular level.

Ground-based teleoperation of robot manipulators for on-orbit servicing of spacecraft represents an example of high-payoff, high-risk operations that are challenging to perform due to high latency communications, with telemetry time delays of several seconds. In these scenarios, confidence of operating without failure is paramount. We report the development of an Interactive Planning and Supervised Execution (IPSE) system that takes advantage of accurate 3D reconstruction of the remote environment to enable operators to plan motions in the virtual world, evaluate and adjust the plan, and then supervise execution with the ability to pause and return to the planning environment at any time. We report the results of an experimental evaluation of a representative on-orbit telerobotic servicing task from NASA’s upcoming OSAM-1 mission to refuel a satellite in low earth orbit; specifically, to change the robot tool to acquire the fuel supply line and then to insert it into the satellite fill/drain valve. Results of a pilot study show that the operators preferred, and were more successful with, the IPSE system when compared to a conventional teleoperation implementation.