First-in-human real-time AI-assisted instrument deocclusion during augmented reality robotic surgery

The integration of Augmented Reality (AR) into daily surgical practice is withheld by the correct registration of pre-operative data. This includes intelligent 3D model superposition whilst simultaneously handling real and virtual occlusions caused by the AR overlay. Occlusions can negatively impact surgical safety and as such deteriorate rather than improve surgical care. Robotic surgery is particularly suited to tackle these integration challenges in a stepwise approach as the robotic console allows for different inputs to be displayed in parallel to the surgeon. Nevertheless, real-time de-occlusion requires extensive computational resources which further complicates clinical integration. This work tackles the problem of instrument occlusion and presents, to the authors' best knowledge, the first-in-human on edge deployment of a real-time binary segmentation pipeline during three robot-assisted surgeries: partial nephrectomy, migrated endovascular stent removal, and liver metastasectomy. To this end, a state-of-the-art real-time segmentation and 3D model pipeline was implemented and presented to the surgeon during live surgery. The pipeline allows real-time binary segmentation of 37 non-organic surgical items, which are never occluded during AR. The application features real-time manual 3D model manipulation for correct soft tissue alignment. The proposed pipeline can contribute towards surgical safety, ergonomics, and acceptance of AR in minimally invasive surgery. This works presents the first-in-human edge deployment of a real-time AI-enabled augmented reality (AR) pipeline in robotic surgery. The application uses a binary segmentation model to effectively identify over 37 classes of non-organic items in the surgical scene, and uses this information to create an overlay visualization, solving the instrument occlusion problem, and preventing the possibly hazardous situation this implies, as well as adding a sense of depth to the AR. The solution is used during three real surgeries and segmentation results, application performance as well as qualitative surgical feedback are discussed.###image