A new hyperspectral imaging system has been designed for integration in the operating room to detect anatomical tissues hardly noticed by the surgeon’s naked eye. This LCTF-based spectral imaging system is operative over visible and near infrared range (400-1100 nm). After spectral calibration and spatial registration, the tricky process consists in reducing the huge amount of acquired data and removing redundancy without losing valuable information. Band transformation and selection methods are applied on both labeled and unlabeled tissues to extract relevant information to be displayed on surgeon’s RGB monitor. Visualization processing involving global and local contrast enhancement is then performed. To provide a reference for evaluation, surgeon’s perception of the scene is also simulated based on retina cell spectral responses. Experiments on pig ureter hyperspectral datasets reveal that band selection methods are the most effective on this type of intervention, providing sharp interpretation and accurate visualization of the biological tissues.
An iterative approach to building a surrogate-driven motion model exclusively from cone-beam CT projections is presented. At each iteration the motion model is updated via an analytical expression derived from an optical flow-based approach, with corresponding improvements in the motion compensated reconstruction. The differences between the actual and estimated motion, as seen in the projections, are incorporated into a modified CBCT reconstruction. The correlations between these differences and the surrogate signals used in the motion model are also taken into account in determining the motion model updates. The updates are then composed with the previous estimate of the motion model and set as the new estimate of the motion model. New updates to this new estimate can then be calculated. The motion model could be used to better understand respiratory motion immediately prior to a fraction of radiotherapy treatment, or to monitor key regions of interest during tracked treatments. This method would also be a promising candidate to adapt an older model built during planning to the day of treatment. The local, voxel-wise updates to the model can account for large inter-fraction changes, specific to the day of treatment. Results on a simulated case are presented, derived from an actual patient dataset undergoing radiotherapy treatment for lung cancer. With the fitted motion, simulated projections of the animated patient volume were seen to be more similar to the actual projections than projections of the static patient volume. When compared with the actual motion, the mean L2-error over the entire patient was reduced to 0.46 mm.
Minimally invasive surgery is an important breakthrough in the domain of medicine. Not only does it improve the quality of surgery, but the underlying digitization also provides invaluable information that opens up many possibilities for teaching, assistance during difficult cases, and quality evaluation. For instance, with a well-organized database, professors are one click away from showing and comparing various surgical procedures in their classes; surgeons can also retrieve and observe a video segment of a specific surgical task performed by another surgeon in varying conditions. However, to the best of our knowledge, database organization is done manually by experts. Considering the large number of surgical videos recorded, manual annotation is a tedious task. In this paper, we take the first step towards automatic surgical database organization by introducing the laparoscopic video classification problem, which consists of automatically identifying the type of abdominal surgery performed in a video. In spite of the visual challenges of such videos, such as blank frames, rapid movement, and sometimes incomplete recording, we show that we can rely on visual features alone to classify the videos with high accuracy. We use kernel Support Vector Machines (SVMs) for this classification task and compare their performance on different types of visual features. We also show that the result can be improved by combining the visual features using Multiple Kernel Learning approach. The classification pipeline demonstrates a classification accuracy of 91.39
Liver resection is the main curative option for liver metastases. While this offers a 5-year survival rate of 50
In the medical field, there exists some surgical simulators and training platforms that have been developed for training novice surgeons in order to improve their surgical skills and for performing preoperative planning. In this paper we present a haptic platform for surgical needle insertion training gestures. It uses passive brakes based on Electro-Rheological (ER) fluids to provide a safe and realistic physical feedback to the physician. To achieve this objective, a prototype has been built, its kinematic model has been obtained and experimentally validated. The modelling, the bandwidth analysis and the force control scheme of the platform are also presented.
Tracking and body pose estimation of clinical staff have several applications in the analysis of surgical workflow, such as radiation monitoring, surgical activity recognition and the study of ergonomics. The operating room is, however, a very complex environment for visual tracking due to frequent illumination changes, clutter, similar color of clinicians’ scrubs and limited sensor positioning. Furthermore, several applications, such as radiation monitoring, require consistent and accurate body part tracking over defined periods of time, which is a challenging task in the aforementioned conditions. In this paper, we tackle the problem of pose estimation in the interventional room. We also propose a method to consistently track upper body parts in short sequences by using RGBD data and discrete Markov Random Field (MRF) optimization over the complete set of frames. The proposed MRF energy formulation enforces both body kinematic and temporal constraints in order to cope with the natural ambiguities of tracking and with the frequent failure of the underlying depth-based body part detector in such conditions. We evaluate our approach quantitatively on seven manually-annotated sequences recorded in the interventional room and show that it can consistently track the upper-body of persons present in the room.
Image fusion of liver 2D X-ray images and pre or peri-operative 3D reconstructions can add valuable contextual information during image guided interventions. Such image fusion requires 2D/3D registration. In abdominal interventions, such as TACE of liver tumors, the initial alignment may be invalidated by e.g. breathing motion. We present a method that maintains the alignment between 3D Rotational Angiography (3DRA) and 2D X-ray, using the catheter position. To this end, we use the catheter in the 2D X-ray and the blood vessels in the 3DRA, then fuse 2D/3D using the knowledge that the catheter is inside the vessels. The registration is performed in two steps: First, we use a shape constraint to determine the most likely catheter positions inside the blood vessel tree. Next, we perform a rigid registration and take the best transformation over all previous selected catheter positions. The method is evaluated on phantom, clinical and simulated data.
We present a new system to acquire and reconstruct 3D freehand ultrasound volumes from arbitrary 2D image acquisitions over time. Motion artifacts are significantly reduced with a novel gating approach which correlates pulse oximetry data with Doppler ultrasound. The reconstruction problem is split into a ray-based sample selection on a per-scanline basis and a backward algorithm which is based on the concept of normalized convolution. We introduce an adaptive derivation of time-domain interpolation from the correlated pulse-oximetry and Doppler signals as well as an ellipsoid kernel size for spatial interpolation based on the physical resolution of the ultrasound data. We compare pulse-oximetry to classical ECG gating and further show the suitability of our normalized pulse signal for 3D+T reconstructions. The ease of use of the setup without the need of uncomfortable triggering via ECG provides the ability to use 3D+T ultrasound in every day clinical practice.
Compounding 2D ultrasound sweeps into 3D volumes is, due to its cost- and time-efficiency, of great clinical significance in both diagnostic and interventional imaging. However, today's algorithms restrict the sweeps to have homogeneous pressure and a linear trajectory, which limits their use in clinical applications such as breast or musculoskeletal ultrasound where artifacts occur due to soft and uneven surfaces. In this work, we present two techniques to resolve those restrictions by using an orientation-driven approach, first compensating for probe pressure changes and then resolving ambiguities in regions, where multiple ultrasound frames from different acoustic windows overlap. After clustering incoming frames by orientation, we determine the final voxel intensities based on per-pixel uncertainty information. Qualitative and quantitative evaluation of our methods shows that these techniques provide reconstructions of superior quality for ultrasound sweeps of inhomogeneous pressure and twisted trajectories. Furthermore, we propose optimizations in the implementation of these techniques towards real-time applications, interactively updating and refining the reconstructed volume.
The rise of intraoperatively available information threatens to outpace our abilities to process data and thus cause informational overload. Context-aware systems, filtering information to match the current situation in the OR, will be necessary to reap all benefits of integrated and computerized surgery. To interpret surgical situations, such systems need a robust set of knowledge to make sense of intraoperative measurements. Building on our own ontology for laparoscopy, we formalized the workflow of laparoscopic adrenalectomies, cholecystectomies and pancreatic resections and developed a novel, rule-based situation interpretation algorithm based on OWL and SWRL to recognize phases of these surgeries. The system was evaluated on ground truth data from 19 manually annotated surgeries with an average recognition rate of 89%.
Instrument localization and tracking is an important challenge for advanced computer assisted techniques in minimally invasive surgery and image-based solutions to instrument localization can provide a non-invasive, low cost solution. In this study, we present a novel algorithm capable of recovering the 3D pose of laparoscopic surgical instruments combining constraints from a classification algorithm, multiple point features, stereo views (when available) and a linear motion model to robustly track the tool in surgical videos. We demonstrate the improved robustness and performance of our algorithm with optically tracked ground truth and additionally qualitatively demonstrate its performance on in vivo images.
Approximately 20–30% of patients with focal epilepsy are medically refractory and may be candidates for curative surgery. Stereo EEG is the placement of multiple depth electrodes into the brain to record seizure activity and precisely identify the area to be resected. The two important criteria for electrode implantation are accurate navigation to the target area, and avoidance of critical structures such as blood vessels. In current practice neurosurgeons have no assistance in the planning of the electrode trajectories. To provide assistance a real-time solution was developed that first identifies the potential entry points by analysing the entry-angle, then computes the associated risks for trajectories starting from these locations. The entry angle, the total length of the trajectory and distances to critical structures are presented in an interactive way that is integrated with standard electrode placement planning tools and advanced visualisation. We show that this improves the planning of intracranial implantation, with safer trajectories in less time.
In conventional prostate biopsy for cancer diagnosis, the 2D nature of ultrasound (US) guidance limits targeting accuracy and does not allow a 3D record of core locations. Several research groups are investigating the use of an electromagnetically tracked US transducer to reconstruct a volumetric scan. Unfortunately, the tracking measurements contain significant errors that affect spatial accuracy. We propose a new filter-based framework of speckle tracking for enchantment of prostate volume reconstruction based on speckle/noise extraction and provide its theoretical basis. A gamma multiplicative noise model is considered and a probability patch-based non-local means (PPB-NLM) filter is used for the task of speckle extraction. The spatial variation of the beam profile is also incorporated using a linear regression model of the beam. Validation tests are first performed on tissue samples obtained ex vivo using a linear motor stage and an optical tracker as gold standards. Further validation is performed on the gastrocnemius muscle in vivo. We then demonstrate the performance of the tracking system on prostate scans obtained in vivo. The results show that the proposed approach produces visually continuous anatomical boundaries in reconstructed 3D US volumes of the prostate.
Spine needle injections are widely applied to alleviate pain and to remove nerve sensation through analgesia and anesthesia. Currently, spinal injections are performed using either no image guidance or modalities that expose the patient to ionizing radiation such as fluoroscopy or computed tomography (CT). Ultrasound (US) is being investigated as an alternative as it is a non-ionizing and more accessible image modality. An inherent challenge to US imaging of the spine is the acoustic shadows created by the bony structures of the vertebrae limiting visibility. It is possible to enhance the anatomical information in US through its fusion with a pre-operative CT. In this manuscript we propose a clinical feasibility study involving a novel registration pipeline to align CT and US images of the spine. This pipeline involves automatic global and multi-vertebrae registration. We evaluate the proposed methodology on five clinical data sets. The proposed method is able to register the data sets from initial misalignments of up to 25 mm, with a mean TRE of 1.17 mm, sufficient for many spine needle interventions.
In this paper, we present a novel method dealing with the identification of boundary conditions of a deformable organ, a particularly important step for the creation of patient-specific biomechanical models of the anatomy. As an input, the method requires a set of scans acquired in different body positions. Using constraint-based finite element simulation, the method registers the two data sets by solving an optimization problem minimizing the energy of the deformable body while satisfying the constraints located on the surface of the registered organ. Once the equilibrium of the simulation is attained (i.e. the organ registration is computed), the surface forces needed to satisfy the constraints provide a reliable estimation of location, direction and magnitude of boundary conditions applied to the object in the deformed position. The method is evaluated on two abdominal CT scans of a pig acquired in flank and supine positions. We demonstrate that while computing a physically admissible registration of the liver, the resulting constraint forces applied to the surface of the liver strongly correlate with the location of the anatomical boundary conditions (such as contacts with bones and other organs) that are visually identified in the CT images.
Identifying and recognizing the workflow of surgical interventions is a field of growing interest. Several methods have been developed to identify intra-operative activities, detect common phases in the surgical workflow and combine the gained knowledge into Surgical Process Models. Numerous applications of this knowledge are conceivable, from semi-automatic report generation, teaching and objective surgeon evaluation to context-aware operating rooms and simulation of interventions to optimize the operating room layout. In this work we propose a method to utilize random decision forests to detect surgical workflow phases based on instrument usage data and other, easily obtainable measurements. While decision forests have become a very versatile and popular tool in the field of medical image analysis, this is to the best of our knowledge its first application to surgical workflow analysis. Our method is in principle suitable for online usage and does not rely on an explicit model or a strict temporal relationship between observations. With their structure, random forests are inherently suited for multi-class detection and therefore for detection of workflow phases. Due to the transparent nature of random forests, additional information may also be obtainable in parallel to the phase detection.
Acute coronary syndrome represents a leading cause of death. Events are triggered by rupture of atheromatic plaques, as a result of disruption of the overlying fibrous cap. Pathological studies have shown that cap thickness is a critical component of plaque stability. Therefore, assessment of fibrous cap thickness could be a valuable tool for estimating the risk of future events. To aid preoperative planning and peri-operative decision making, intracoronary optical coherence tomography imaging can provide very detailed information about arterial wall structure. However, manual interpretation of the images is laborious, subject to variability, and therefore not always sufficiently reliable for immediate decision of treatment. We present a novel semi-automatic computerized interventional imaging tool to quantify coronary fibrous cap thickness in optical coherence tomography. The most challenging issue when estimating cap thickness is caused by the diffuse nature of the anatomical abluminal interface to be detected. Our method can successfully extract the fibrous cap contours using a robust dynamic programming framework based on a geometrical a priori. Validated on a dataset of 90 images from 11 patients, our method provided a good agreement for minimum cap thickness with the reference tracings performed by a medical expert (35.7 ±33.3 μm, R=.68) and was similar to inter-observer reproducibility (35.2 ±33.1 μm, R=.66), while being significantly faster and fully reproducible. This tool demonstrated promising performances and could potentially be used for online identification of high risk-plaques.
Intraoperative MRI is a powerful modality for acquiring structural and functional images of the brain to enable precise image-guided neurosurgery. In this paper, we propose a novel method for simulating main magnetic field inhomogeneity maps during intraoperative MRI-guided neurosurgery. Our method relies on an air-tissue segmentation of intraoperative patient specific data, which is used as an input to a subsequent field simulation step. The generated simulation can then be used to enhance the precision of image-guidance. We report results of our method on 12 patient datasets acquired during image-guided neurosurgery for anterior lobe resection for surgical management of focal temporal lobe epilepsy. We find a close agreement between the field inhomogeneity maps acquired as part of the imaging protocol and the simulated field inhomogeneity maps generated by the proposed method.
In computer-aided interventions, the visual feedback of the doctor is vital. Enhancing the relevant object will help for the perception of this feedback. In this paper, we present a learning-based labeling of the surgical scene using a depth camera (comprised of RGB and depth range sensors). The depth sensor is used for background extraction and Random Forests are used for segmenting color images. The end result is a labeled scene consisting of surgeon hands, surgical instruments and background labels. We evaluated the method by conducting 10 simulated surgeries with 5 clinicians and demonstrated that the approach provides surgeons a dissected surgical scene, enhanced visualization, and upgraded depth perception.
Intramedullary nailing is the surgical procedure mostly used in fracture reduction of the tibial and femoral shafts. Following successful insertion of the nail into the medullary canal, it must be fixed by inserting screws through its proximal and distal locking holes. Prior to distal locking of the nail, surgeons must position the C-arm device and patient leg in such a way that the nail holes appear as circles in the X-ray image. This is considered a ‘trial and error’ process, is time consuming and requires many X-ray shots. We propose an augmented reality application that visually depicts to the surgeon two ‘augmented’ circles, their centers lying on the axis of the nail hole, making it visible in space. After an initial X-ray image acquisition, real-time video guidance allows the surgeon to superimpose the ‘augmented’ circles by moving the patient leg; the result being nail holes appearing as circles. Our nail pose recovery was evaluated on 1000 random trials and we consistently recovered the nail angulation within 2.76 ± 1.66°. Lastly, in a preclinical experiment involving 7 clinicians, we demonstrated that in over 95
