Novel Application of Real-Time Editable 3D-reconstructed Imaging in a Mixed-Reality Holographic Overlay Using a Video Pass-through Head-Mounted Display – A Pathway to future Navigation in Thoracic Surgery?

Abstract Introduction: 3D reconstructions of widely available high-resolution imaging is only rarely used for preprocedural assessment in thoracic surgery. However, it is a promising tool that aims to improve patient-specific treatment planning. Increasingly available mixed reality hardware based on video pass-through technology enables to project image data as a hologram onto the patient. We describe the novel method of real-time 3D surgical planning in a mixed reality setting presenting two representative cases. Material and Methods: A mixed reality system was set-up using a high-performance workstation running a video pass-through based head-mounted display. Image data of computer tomography was imported and visually customized through live editing. The image-based hologram was projected onto the patient highlighting the regions of interest. Results: Two oncological cases were selected, both presenting large tumor masses in the thoracic cavity. We aligned 3D holographic image data onto the patient allowing us to investigate the relationship between anatomical structures and their respective body position. Discussion and Outlook: The exploration of holographic overlay proved to be promising in improving preprocedural surgical planning, particularly on complex oncological tasks in the thoracic surgery area. A pilot study on approach planning shall therefore be conducted. Ongoing technological progresses of extended reality hardware and intelligent software features will most likely enhance applicability and range of use in surgical fields within the near future.