PurposeThe discrepancy of continuously decreasing opportunities for clinical training and assessment and the increasing complexity of interventions in surgery has led to the development of different training and assessment options like anatomical models, computer-based simulators or cadaver trainings. However, trainees, following training, assessment and ultimately performing patient treatment, still face a steep learning curve.MethodsTo address this problem for C-arm-based surgery, we introduce a realistic radiation-free simulation system that combines patient-based 3D printed anatomy and simulated X-ray imaging using a physical C-arm. To explore the fidelity and usefulness of the proposed mixed-reality system for training and assessment, we conducted a user study with six surgical experts performing a facet joint injection on the simulator.ResultsIn a technical evaluation, we show that our system simulates X-ray images accurately with an RMSE of 1.85mm compared to real X-ray imaging. The participants expressed agreement with the overall realism of the simulation, the usefulness of the system for assessment and strong agreement with the usefulness of such a mixed-reality system for training of novices and experts. In a quantitative analysis, we furthermore evaluated the suitability of the system for the assessment of surgical skills and gather preliminary evidence for validity.ConclusionThe proposed mixed-reality simulation system facilitates a transition to C-arm-based surgery and has the potential to complement or even replace large parts of cadaver training, to provide a safe assessment environment and to reduce the risk for errors when proceeding to patient treatment. We propose an assessment concept and outline the steps necessary to expand the system into a test instrument that provides reliable and justified assessments scores indicative of surgical proficiency with sufficient evidence for validity.

Chirurgische Simulatoren kommen neben klassischen Tiermodellen und Humanpräparaten zunehmend als attraktive Alternative zur klinischen Ausbildung zum Einsatz. Chirurgische Simulationstechnologie ist typischerweise darauf ausgelegt, chirurgisch-technische Fertigkeiten zu vermitteln („task trainer“). Das Simulatortraining in der Chirurgie beschränkt sich daher auf die individuelle Ausbildung des Chirurgen und berücksichtigt die Beteiligung des restlichen OP-Teams nicht. Das Ziel des Projektes „Assessment and Training of Medical Experts based on Objective Standards“ (ATMEOS) ist die Entwicklung einer immersiven simulierten OP-Umgebung, die es ermöglicht, multidisziplinäre OP-Teams zu trainieren und deren Leistung unter verschiedenen Bedingungen zu bewerten. Hierbei findet ein Mixed-reality-Ansatz Anwendung, der ein synthetisches Patientenmodell, reale chirurgische Instrumente und strahlungsfreies virtuelles Röntgen zur Simulation von Wirbelsäuleneingriffen kombiniert. In vorangegangenen Forschungsarbeiten wurde das Konzept hinsichtlich Realitätstreue, Plausibilität und Immersivität evaluiert. In der aktuellen Forschung werden Metriken zur Bewertung technischer und nichttechnischer Fähigkeiten entwickelt und evaluiert. Ziel ist, in der simulierten OP-Umgebung multidisziplinäre OP-Teams bei minimalinvasiven Eingriffen an der Wirbelsäule zu beobachten sowie die Leistung der einzelnen Teammitglieder und des gesamten Teams objektiv zu bewerten. Zusätzlich können die Effektivität von Trainingsmethoden und Operationstechniken oder erfolgskritische Faktoren, z. B. der Umgang mit Krisensituationen, in der kontrollierten Umgebung erfasst und objektiv bewertet werden.

Minimally invasive surgeries (MISs) are gaining popularity as alternatives to conventional open surgeries. In thoracoscopic scoliosis MIS, fluoroscopy is used to guide pedicle screw placement and to visualise the effect of the intervention on the spine curvature. However, cosmetic external appearance is the most important concern for patients, while correction of the spine and achieving coronal and sagittal trunk balance are the top priorities for surgeons. The authors present the feasibility study of the first intra-operative assistive system for scoliosis surgery composed of a single RGBD camera affixed on a C-arm which allows visualising in real time the surgery effects on the patient trunk surface in the transverse plane. They perform three feasibility experiments from simulated data based on scoliotic patients to live acquisition from non-scoliotic mannequin and person, all showing that the proposed system accuracy is comparable with scoliotic surface reconstruction state of art.

Medical Mixed Reality helps surgeons to contextualize intraoperative data with video of the surgical scene. Nonetheless, the surgical scene and anatomical target are often occluded by surgical instruments and surgeon hands. In this paper and to our knowledge, we propose a multi-layer visualization in Medical Mixed Reality solution which subtly improves a surgeon's visualization by making transparent the occluding objects. As an example scenario, we use an augmented reality C-arm fluoroscope device. A video image is created using a volumetric-based image synthesization technique and stereo-RGBD cameras mounted on the C-arm. From this synthesized view, the background which is occluded by the surgical instruments and surgeon hands is recovered by modifying the volumetric-based image synthesization technique. The occluding objects can, therefore, become transparent over the surgical scene. Experimentation with different augmented reality scenarios yield results demonstrating that the background of the surgical scenes can be recovered with accuracy between 45%-99%. In conclusion, we presented a solution that a Mixed Reality solution for medicine, providing transparency to objects occluding the surgical scene. This work is also the first application of volumetric field for Diminished Reality/ Mixed Reality.

The discrepancy of continuously decreasing clinical training opportunities and increasing complexity of interventions in surgery has led to the development of different training options like anatomical models, computer-based simulators or cadaver trainings. However, trainees, following this training and ultimately performing patient treatment, still face a steep learning curve. To address this problem for C-arm based surgery, we introduce a realistic radiation-free simulation system that combines patient-based 3D printed anatomy and simulated X-ray imaging using a physical C-arm. This mixed reality simulation system facilitates a transition to C-arm based surgery and has the potential to complement or even replace large parts of cadaver training and to reduce the risk for errors when proceeding to patient treatment. In a technical evaluation, we show that our system simulates X-ray images accurately with an RMSE of 1.85 mm compared to real X-ray imaging. To explore the fidelity and usefulness of the proposed mixed reality system for training and assessment, we conducted a user study. Six surgical experts performed a facet joint injection on the simulator and rated aspects of the system on a 5-point Likert scale. They expressed agreement with the overall realism of the simulation and strong agreement with the usefulness of such a mixed reality system for training of novices and experts.

Calibration and registration are the first steps for augmented reality and mixed reality applications. In the medical field, the calibration between an RGB-D camera and a C-arm fluoroscope is a new topic which introduces challenges.

PURPOSE:In many orthopedic surgeries, there is a demand for correctly placing medical instruments (e.g., K-wire or drill) to perform bone fracture repairs. The main challenge is the mental alignment of X-ray images acquired using a C-arm, the medical instruments, and the patient, which dramatically increases in complexity during pelvic surgeries. Current solutions include the continuous acquisition of many intra-operative X-ray images from various views, which will result in high radiation exposure, long surgical durations, and significant effort and frustration for the surgical staff. This work conducts a preclinical usability study to test and evaluate mixed reality visualization techniques using intra-operative X-ray, optical, and RGBD imaging to augment the surgeon's view to assist accurate placement of tools.METHOD:We design and perform a usability study to compare the performance of surgeons and their task load using three different mixed reality systems during K-wire placements. The three systems are interventional X-ray imaging, X-ray augmentation on 2D video, and 3D surface reconstruction augmented by digitally reconstructed radiographs and live tool visualization.RESULTS:The evaluation criteria include duration, number of X-ray images acquired, placement accuracy, and the surgical task load, which are observed during 21 clinically relevant interventions performed by surgeons on phantoms. Finally, we test for statistically significant improvements and show that the mixed reality visualization leads to a significantly improved efficiency.CONCLUSION:The 3D visualization of patient, tool, and DRR shows clear advantages over the conventional X-ray imaging and provides intuitive feedback to place the medical tools correctly and efficiently.

INTRODUCTION:In the modern day operating room, the surgeon performs surgeries with the support of different medical systems that showcase patient information, physiological data, and medical images. It is generally accepted that numerous interactions must be performed by the surgical team to control the corresponding medical system to retrieve the desired information. Joysticks and physical keys are still present in the operating room due to the disadvantages of mouses, and surgeons often communicate instructions to the surgical team when requiring information from a specific medical system. In this paper, a novel user interface is developed that allows the surgeon to personally perform touchless interaction with the various medical systems, switch effortlessly among them, all of this without modifying the systems' software and hardware.METHODS:To achieve this, a wearable RGB-D sensor is mounted on the surgeon's head for inside-out tracking of his/her finger with any of the medical systems' displays. Android devices with a special application are connected to the computers on which the medical systems are running, simulating a normal USB mouse and keyboard. When the surgeon performs interaction using pointing gestures, the desired cursor position in the targeted medical system display, and gestures, are transformed into general events and then sent to the corresponding Android device. Finally, the application running on the Android devices generates the corresponding mouse or keyboard events according to the targeted medical system.RESULTS AND CONCLUSION:To simulate an operating room setting, our unique user interface was tested by seven medical participants who performed several interactions with the visualization of CT, MRI, and fluoroscopy images at varying distances from them. Results from the system usability scale and NASA-TLX workload index indicated a strong acceptance of our proposed user interface.

X-ray is still the essential imaging for many minimally-invasive interventions. Overlaying X-ray images with an optical view of the surgery scene has been demonstrated to be an efficient way to reduce radiation exposure and surgery time. However, clinicians are recommended to place the X-ray source under the patient table while the optical view of the real scene must be captured from the top in order to see the patient, surgical tools, and the surgical site. With the help of a RGB-D (red-green-blue-depth) camera, which can measure depth in addition to color, the 3D model of the real scene is registered to the X-ray image. However, fusing two opposing viewpoints and visualizing them in the context of medical applications has never been attempted. In this paper, we propose first experiences of a novel inverse visualization technique for RGB-D augmented C-arms. A user study consisting of 16 participants demonstrated that our method shows a meaningful visualization with potential in providing clinicians multi-modal fused data in real-time during surgery.

C-arm fluoroscopes are frequently used during surgeries for intraoperative guidance. Unfortunately, due to X-ray emission and scattering, increased radiation exposure occurs in the operating theatre. The objective of this work is to sensitize the surgeon to their radiation exposure, enable them to check on their exposure over time, and to help them choose their best position related to the C-arm gantry during surgery. First, we aim at simulating the amount of radiation that reaches the surgeon using the Geant4 software, a toolkit developed by CERN. Using a flexible setup in which two RGB-D cameras are mounted to the mobile C-arm, the scene is captured and modeled respectively. After the simulation of particles with specific energies, the dose at the surgeon's position, determined by the depth cameras, can be measured. The validation was performed by comparing the simulation results to both theoretical values from the C-arms user manual and real measurements made with a QUART didoSVM dosimeter. The average error was 16.46% and 16.39%, respectively. The proposed flexible setup and high simulation precision without a calibration with measured dosimeter values, has great potential to be directly used and integrated intraoperatively for dose measurement.

This paper presents the first design of a mirror based RGBD X-ray imaging system and includes an evaluation study of the depth errors induced by the mirror when used in combination with an infrared pattern-emission RGBD camera. Our evaluation consisted of three experiments. The first demonstrated almost no difference in depth measurements of the camera with and without the use of the mirror. The final two experiments demonstrated that there were no relative and location-specific errors induced by the mirror showing the feasibility of the RGBDX-ray imaging system. Lastly, we showcase the potential of the RGBDX-ray system towards a visualization application in which an X-ray image is fused to the 3D reconstruction of the surgical scene via the RGBD camera, using automatic C-arm pose estimation.

Fusing intraoperative X-ray data with real-time video in a common reference frame is not trivial since both modalities have to be acquired from the same viewpoint. The goal of this work is to design a flexible system comprising two RGBD sensors that can be attached to any mobile C-arm, with the objective of synthesizing projective color images from the X-ray source viewpoint. To achieve this, we calibrate the RGBD sensors to the X-ray source with a 3D calibration object. Then, we synthesize the projective color image from the X-ray viewpoint by applying a volumetric-based rendering method. Finally, the X-ray image is overlaid on the projective image without any further registration, offering a multimodal visualization of X-ray and color images. In this paper we present the different steps of development (i.e. hardware setup, calibration and rendering algorithm) and discuss clinical applications for the new video augmented C-arm. By placing X-ray markers on a hand patient and a spine model, we show that the overlay accuracy between the X-ray image and the synthetized image is in average 1.7 mm.

Calibration and registration are the first steps for augmented reality and mixed reality applications. In the medical field, the calibration between an RGB-D camera and a mobile C-arm fluoroscope is a new topic which introduces challenges. In this paper, we propose a precise 3D/2D calibration method to achieve a video augmented fluoroscope. With the design of a suitable calibration phantom for RGB-D/C-arm calibration, we calculate the projection matrix from the depth camera coordinates to the X-ray image. Through a comparison experiment by combining different steps leading to the calibration, we evaluate the effect of every step of our calibration process. Results demonstrated that we obtain a calibration RMS error of 0.54±1.40 mm which is promising for surgical applications. We conclude this paper by showcasing two clinical applications. One is a markerless registration application, the other is an RGB-D camera augmented mobile C-arm visualization.

In computer-aided interventions, the visual feedback of the doctor is vital. Enhancing the relevant object will help for the perception of this feedback. In this paper, we present a learning-based labeling of the surgical scene using a depth camera (comprised of RGB and depth range sensors). The depth sensor is used for background extraction and Random Forests are used for segmenting color images. The end result is a labeled scene consisting of surgeon hands, surgical instruments and background labels. We evaluated the method by conducting 10 simulated surgeries with 5 clinicians and demonstrated that the approach provides surgeons a dissected surgical scene, enhanced visualization, and upgraded depth perception.

We present the idea of a user interface concept, which resolves the challenges involved in the control of angiographic C-arms for their constant repositioning during interventions by either the surgeons or the surgical staff. Our aim is to shift the paradigm of interventional image acquisition workflow from the traditional control device interfaces to ‘desired-view’ control. This allows the physicians to only communicate the desired outcome of imaging, based on simulated X-rays from pre-operative CT or CTA data, while the system takes care of computing the positioning of the imaging device relative to the patient’s anatomy through inverse kinematics and CT to patient registration. Together with our clinical partners, we evaluate the new technique using 5 patient CTA and their corresponding intraoperative X-ray angiography datasets.

We present a novel algorithm for the registration of multiple temporally related point sets. Although our algorithm is derived in a general setting, our primary motivating application is coronary tree matching in multi-phase cardiac spiral CT. Our algorithm builds upon the fast, outlier-resistant Coherent Point Drift (CPD) algorithm, but incorporates temporal consistency constraints between the point sets, resulting in spatiotemporally smooth displacement fields. We preserve the speed and robustness of the CPD algorithm by using the technique of separable surrogates within an EM (Expectation-Maximization) optimization framework, while still minimizing a global registration cost function employing both spatial and temporal regularization. We demonstrate the superiority of our novel temporally consistent group-wise CPD algorithm over a straightforward pair-wise approach employing the original CPD algorithm, using coronary trees derived from both simulated and real cardiac CT data. In all the tested configurations and datasets, our method presents lower average error between tree landmarks compared to the pairwise method. In the worst case, the difference is around few micrometers but in the better case, our method divides by two the error from the pairwise method. This improvement is especially important for a dataset with numerous outliers. With a fixed set of parameter that has been tuned automatically, our algorithm yields better results than the original CPD algorithm which shows the capacity to register without a priori information on an unknown dataset.

Coronary artery (CA) aneurysms may complicate Kawasaki disease (KD) in 30% of untreated patients and up to 5% of those treated with IVIG. Despite new trends to use CT-angiography and MRA to image CA aneurysms, selective catheter-based angiography remains widely utilized especially when percutaneous intervention is contemplated. Depending on the complexity and spatial deformity of the aneurysm biplane projections may not be satisfactory during selective angiography. Up to 2 or 3 additional 2D injections/projections may be required for accurate determination of stenotic entry/exit sites to the aneurysms. Currently, the 3D reconstruction techniques from 2D angiograms are not fast enough for online live rendering. When feasible, such reconstructions remain limited to 2 or 3 views. To perform 3D online reconstruction for optimal visualization of target lesions, on had 3D manipulation to suggest the best 2D views for the intervention, and to limit the number of projections thus avoiding unnecessary X-ray exposure. We used the Shape-from-Silhouette method on silhouettes of coronary arteries obtained from segmented angiographies. This method creates projective cones from 2D silhouettes to create 3D reconstruction by intersection of these cones. We then validated of our methodology from a model of LCA created from a healthy patient CT-scan on which aneurysms have been numerically added. Seven coronary artery silhouettes corresponding to the most used angiographic views have been generated. The reconstruction from 3 views gives a shape similar to the gold standard. Additional visual enhancement with increasing number of views varies according to the target CA segment. Tabled 1 The LCA model shows the possibility to reduce the necessary number of views to 3-4 views. Reconstruction speed is 30 sec with 3 views and 68 sec with 7 views. Our algorithm's speed is favorable for "live" 3D CA reconstruction. Virtual 3D navigation enables the physician to select the optimal projection according the aneurysm's location for an eventual intervention and follow-up studies. We are currently applying our methodology to actual selective CA angiograms with a goal to incorporate our algorithm in the clinical practice as a strategy to optimize the diagnosis and to minimize patients' exposure to ionizing radiation.

This paper presents a new application of the shape from silhouette (SFS) method for the 3D reconstruction of coronary arteries with Kawasaki disease from angiographic images. The silhouettes of the arteries were first segmented from the angiographic images using the Frangi filter followed by thresholding then a morphologic erosion step. Finally, a volume-based reconstruction approach was performed to infer the visual hull representing the 3D volume of the segmented arteries. The proposed method was first validated on simulated data to determine the optimal number of views that provide a 3D reconstruction with an adequate precision. The sensitivity of the proposed method to calibration errors was also evaluated. Furthermore, a clinical evaluation on four patients with Kawasaki disease showed that using three views in addition to the standard AP and LAT views does not improve the accuracy of the 3D reconstruction. Therefore, this study showed that an online 3D reconstruction of the coronary arteries, requiring less than one minute, during the acquisition could help to rationalize the number of views required for the 3D assessment of aneurysms.