The success of deep brain stimulation (DBS) is dependent on the accurate placement of electrodes in the operating room (OR). However, due to intraoperative brain shift, the accuracy of pre-operative scans and pre-surgical planning are often degraded. To compensate for brain shift, we created a finite element bio-mechanical brain model that updates preoperative images by assimilating intraoperative sparse data from the brain surface or deep brain targets. Additionally, we constructed an artificial neural network (ANN) that leveraged a large number of ventricle nodal displacements to estimate brain shift. The machine learning method showed potential in incorporating ventricle sparse data to accurately compute shift at the brain surface. Thus, in this paper, we propose using this machine learning model to estimate brain atrophy at deep brain targets such as the anterior commissure (AC) and the posterior commissure (PC). The ANN consists of an input layer with nine hand-engineered features, such as the distance between the deep brain target and the ventricle node, two hidden layers and an output layer. This model was trained using eight patient cases and tested on two patient cases.
Ventricular tachycardia (VT) can be treated with catheter ablation therapy, a technique in which catheters are guided into the ventricle and radiofrequency energy is delivered into the myocardial tissue to stop arrhythmic electrical pathways. These procedures are invasive and come with associated risks; therefore, recent efforts have investigated the use of noninvasive proton beam therapy for treatment of VT. In this approach, target regions are identified in a pre-treatment computed tomography scan of the left ventricle followed by proton beam ablation therapy. The effects of beam ablation therapy in myocardial tissue can be characterized using imaging, electroanatomic mapping, and histology. These data are also important for determining the appropriate dose for effective treatment of VT while minimizing collateral damage to surrounding healthy tissue. Studies conducted to date demonstrate that proton beam ablation is a promising new approach for treatment of VT.
Breast conserving surgery is a common treatment option for early-stage breast cancer, and it relies on complete tumor excision such that no residual cancer is left in the resection cavity. However, the use of preoperative imaging to inform excision is compromised by intraoperative deformations that change the location, volume, and shape of the tumor compared to the imaging configuration. For intra-procedural guidance specifically, incision and retraction alter the tumor presentation and geometry. Being able to compensate for retraction deformations intraoperatively may increase the utility of image guidance technologies. In this work, a breast retraction phantom and deformation modeling approach are developed to explore the potential of modeling retraction for image guidance during BCS. Surface and subsurface beads were embedded in a realistic silicone breast phantom, and CT images were acquired in undeformed and retracted states. A reconstructive, sparse-data registration method was used to model retraction. Modeling accuracy was evaluated by comparing model-predicted and ground-truth bead displacements. The average surface bead registration error after retraction modeling in a region of interest was 0.5 ± 0.1 mm (maximum 0.5 mm). The average subsurface bead registration error in a region of interest was 1.2 ± 0.6 mm (maximum 2.6 mm). A biomechanical modeling method that includes retraction may improve the accuracy of image guidance for breast conserving surgery, but more work is needed to evaluate its utility.
Efficient and accurate segmentation of the rectum in images acquired with a low-field (58-74mT), prostate Magnetic Resonance Imaging (MRI) scanner may be advantageous for MRI-guided prostate biopsy and focal treatment guidance. However, automated rectum segmentation on low-field MRI images is challenging due to spatial resolution and signal-to-Noise Ratio (SNR) constraints. This study aims to develop a deep learning model to automatically segment the rectum in a low-field MRI prostate image. 132, 3D images from ten patients were assembled. A 3D, U-Net model with the input matrix size 120×120×40 voxels was trained to detect and segment the rectum. The 3D U-Net can learn and integrate the relative information between adjacent MRI slices, which can enforce 3D patterns such as rectal wall smoothness and thus compensate for slice-to-slice variability in SNR and rectal boundary fuzziness. Contrast stretching, histogram equalization, and brightness enhancement were also investigated and applied to normalize intra- and inter- image intensity heterogeneity. Data augmentation methods such as elastic deformation, flipping, rotation, and scaling were also applied to reduce the risk of overfitting in model training. The model was trained and tested using a 4-fold cross-validation method with 3:1:2 split for training, validation, and testing. Study results show that the mean intersection over union score (IOUs) is 0.63 for the rectum on the testing dataset. Additionally, visual examination suggests that the displacement between the centroids of the ground truth and inferred volumetric segmentations is less than 3mm. Thus, this study demonstrates that (1) a 3D U-Net model can effectively segment the rectum on low-field MRI scans and (2) applying image processing and data augmentation can boost model performance.
Mass Spectrometry Imaging (MSI) is a powerful tool capable of visualizing molecular patterns to identify disease markers in tissue analysis. However, data analysis is computationally heavy and currently time-consuming as there is no single platform capable of performing the entire preprocessing, visualization, and analysis pipeline end-to-end. Using different software tools and file formats required for such tools also makes the process prone to error. The purpose of this work is to develop a free, open-source software implementation called “Visualization, Preprocessing, and Registration Environment” (ViPRE), capable of end-to-end analysis of MSI data. ViPRE was developed to provide various functionalities required for MSI analysis including data import, data visualization, data registration, Region of Interest (ROI) selection, spectral data alignment and data analysis. The software implementation is offered as an open-source module in 3D Slicer, a medical imaging platform. It is also designed for flexibility and usability throughout the user experience. ViPRE was tested using sample MSI data to evaluate the computational pipeline, with the results showing successful implementation of its functionalities and end-to-end usage. A preliminary usability test was also performed to assess user experience, with findings showing positive results. ViPRE aspires to satisfy the need for a single-stop comprehensive interface for MSI data analysis. The source code and documentation will be made publicly available.
Haptic devices allow touch-based information transfer between humans and their environment. In minimally invasive surgery, a human teleoperator benefits from both visual and haptic feedback regarding the interaction forces between instruments and tissues. In this talk, I will discuss mechanisms for stable and effective haptic feedback, as well as how surgeons and autonomous systems can use visual feedback in lieu of haptic feedback. For haptic feedback, we focus on skin deformation feedback, which provides compelling information about instrument-tissue interactions with smaller actuators and larger stability margins compared to traditional kinesthetic feedback. For visual feedback, we evaluate the effect of training on human teleoperators’ ability to visually estimate forces through a telesurgical robot. In addition, we design and characterize multimodal deep learning-based methods to estimate interaction forces during tissue manipulation for both automated performance evaluation and delivery of haptics-based training stimuli. Finally, we describe the next generation of soft, flexible surgical instruments and the opportunities and challenges they present for seeing and feeling in robot-assisted surgery.
Cardiac motion remains a challenge in the treatment of ventricular tachycardia with external beam ablation therapy. Current techniques involve expansion of the treatment area which can lead to unwanted collateral damage. Surrounding healthy tissue could be spared by gating the delivery of the beam to the cardiac cycle. In prior work, we assessed cardiac motion using in vivo fiducial markers and demonstrated that motion would be reduced if treatment were gated to half of the cardiac cycle, approximately corresponding to diastole. In the current work, we extend our prior analysis by quantitatively assessing the optimal gating window for motion reduction in the left ventricle. Motion was assessed in five porcine models with two fiducial clips per animal for a total of ten clips. The minimal cardiac motion occurred when the gating window started at 70% of the cardiac cycle. Without gating, three-dimensional cardiac motion was 7.0 ± 3.9 in x (left/right), 5.3 ± 2.5 in y (anterior/posterior), and 5.6 ± 2.3 in z (superior/inferior) mm. Using an optimal gating window, cardiac motion was 3.1 ± 1.8 in x (left/right), 2.5 ± 1.2 in y (anterior/posterior), and 3.1 ± 1.7 in z (superior/inferior) mm. The percentage reduction in motion with optimal gating was 51 ± 23 in x (left/right), 49 ± 21 in y (anterior/posterior), and 45 ± 24 % in z (superior/inferior). This work demonstrates that gating shows significant promise for reducing the effects of left ventricular motion when treating ventricular tachycardia with external beam ablation therapy.
Vagus Nerve Stimulation (VNS) is an effective technique for treating epilepsy, and it is a promising method to treat many other health conditions, such as depression, cardiovascular disease, chronic pain, diabetes, and others. Due to this wide range of applications, many researchers have developed VNS devices and stimulation techniques over the past decades. However, a common practice is to implant an electrode that has a rather broad stimulation field across the Vagus Nervus (VN) and as a result has limited anatomical specificity and may lead to adverse side effects. The efficacy and breadth of VNS therapy can be improved by selectively modulating only regions associated with a given function. Additionally, enhanced precision should also facilitate uncovering functional vagotopy. In this work, stimulation levels, amount of current injected and electrode configuration are investigated to determine the extent to which activation of the vagus nerve can be precisely controlled. A simple quantitative method to optimize activation is also proposed.
Ultrasound holds promise for use in spinal cord injury cases for both diagnostic and therapeutic purposes. Focused ultrasound applications demand an added threshold of study to ensure the safety and efficacy of the therapy. For optimal treatment outcomes, it is crucial to understand whether relevant structures are being targeting with sufficient energy without damaging neighboring tissue and vasculature. However, it is difficult to predict the expected displacement and pressure profile of the ultrasound wavefront due to challenges with visualizing an acoustic beam in real-time and complex patient-specific anatomy. This challenge is particularly prominent in anatomies with varying medium acoustic properties that cause reflection and distortion of the signal, which is inherent to the composition of the spinal cord and is exacerbated by the formations of injury-induced hematomas. Incorrect placement of focused ultrasound transducers can be detrimental to patient health, specifically if therapeutic ultrasound is used at higher intensities, as the beam propagation can target healthy tissue and important structures that could lead to tissue damage and death. We study how computational tools can be leveraged to aid placement of the transducer using an ultrasound simulation software, Wave 3000 Plus, that allows for the visualization of ultrasound propagation through anatomical structures. By simulating the propagation of ultrasound beams through patient-specific Digital Imaging and Communications in Medicine (DICOM) images, we study computational approaches to determine the optimal placement of devices. In this study, we use in vivo porcine spinal cord images following spinal cord injury (as an example medical use case) to determine if the injury site is being targeted appropriately and to visualize the distribution of pressure throughout the simulation. We demonstrate that Wave 3000 Plus is a viable approach for visualizing ultrasound propagation through patient-specific anatomies.
Because Radial-Probe Endobronchial Ultrasound (RP-EBUS) can provide real-time confirmation of a suspect peripheral nodule situated outside of the airways, it is widely used during bronchoscopy for lung cancer diagnosis. RP-EBUS, however, tends to be difficult to use effectively, without some form of guidance. Previously, we had prototyped a multimodal image-guided bronchoscopy system that provides guidance during both bronchoscopic navigation and RP-EBUS localization. To use the system, the user first generates a guidance plan offline prior to the live procedure. Later, in the surgical suite, the user then employs the image-guided system to perform the desired multimodal RP-EBUS bronchoscopy, driven by the procedure plan. We now validate this system in a series of live studies. As the first set of end-to-end live system studies, we first tested the system in controlled animal studies. Through these studies, we tested the functionality and feasibility of the system prototype over the standard clinical workflow, without the usual risks associated with live patient procedures. Through these studies, we sharpened the workflow for the prototype and improved user interaction. We then tested the refined system over the standard clinical workflow in our University Hospital’s lung cancer management clinic. This study proved the potential of our system for live clinical usage by demonstrating the safety, feasibility, and functionality of our complete system for guiding RP-EBUS bronchoscopy during peripheral nodule diagnosis.
This study investigates a method of time resolved 3D (4D) x-ray imaging of contrast dynamics internal to a vascular structure (e.g. intracranial aneurysm) to enable evaluation of blood flow patterns during an interventional procedure. The proposed method employs repetitive-short-pulse injection of small contrast boluses, rotational x-ray imaging with a C-arm, and retrospectively gated iterative image reconstruction. Under conditions where the passage of each contrast pulse through a vascular region is repeatable and the C-arm rotation is slow compared to the injection cycle, each flow state (spatial distribution of contrast agent at an instant) is imaged at multiple projection angles. After partitioning the projections by flow state, a sequence of 3D volumes corresponding to different states of contrast passage can be reconstructed. Feasibility was demonstrated in a patient-specific 3D-printed aneurysm phantom with 1 Hz simulated cardiac flow waveform. A custom-built power injector was programmed to produce repetitive 100ms injections of iodinated contrast agent upstream of the aneurysm, synchronized to the mid-diastolic phase of the simulated cardiac cycle (1 Hz, 0.4 mL/pulse, 20 pulses, 8 mL total). An interventional C-arm short-scan was performed with 11.3 s rotation time and 27fps frame rate. Modified PICCS reconstruction was used to generate the 4D images. The temporal evolution of contrast agent in the 4D x-ray images was visually similar to the flow patterns observed in MRI imaging and CFD simulation of the same phantom. 95% of the surface deviations between the 4D aneurysm volume and traditional 3D-DSA aneurysm volume were within -0.02 ± 0.24 mm.
Trans-Oral Robotic Surgery (TORS) is an alternative surgery technique used to treat head-and-neck cancer. Compared with conventional surgery, robot assistance allows surgeons to operate within areas with restricted access, such as the oropharynx, reducing the operative morbidity, risk of reconstructive surgery and improving patient outcomes. TORS is a challenging procedure, and intra-operative Ultrasound (US) has the potential to improve anatomy visualization to lessen the cognitive load on surgeons. To date, only intra-oral US has been used in exploratory studies, but intra-oral US can interfere with robot tools. In this study, we assess the feasibility of using transcervical 3D US with TORS: we propose to place the US probe on the patient’s neck to evaluate oropharyngeal anatomy intra-operatively. We also perform the first feasibility study of image registration between transcervical 3D US and Magnetic Resonance Imaging (MRI) for the oropharynx. We collected 3D US and MRI data from five healthy volunteers and four patients with oropharyngeal cancer, and we use a semi-automatic MRI-US registration algorithm to estimate an affine transformation between the two image spaces. The average Target Registration Error (TRE) is 8.26 ± 7.41mm for healthy volunteers and 9.63 ± 5.91mm for patients, and our case studies show that image quality is the key factor for good registration. Our work shows that 3D transcervical US has the clinical potential to enable intraoperative oropharynx imaging and interventional MR guidance during TORS.
Registration of preoperative or intraoperative imaging is necessary to facilitate surgical navigation in spine surgery. After image acquisition, intervertebral motion and spine pose changes can occur during surgery from instrumentation, decompression, physician manipulation or correction. This causes deviations from the reference imaging reducing the navigation accuracy. To evaluate the ability to use the registration between stereovision surfaces in order to account for this intraoperative spine motion through a simulation study. Co-registered CT and stereovision surface data were obtained of a swine cadaver’s exposed lumbar spine in the prone position. Data was segmented and labeled by vertebral level. A simulation of biomechanically bounded motion was applied to each vertebral level to move the prone spine to a new position. A reduced surface data set was then registered level-wise back to the prone spines original position. The average surface to surface distance was recorded between simulated and prone positions. Localized targets on these surfaces were used for a calculation of target registration error. Target registration error increases with distance between surfaces. Movement exceeding 2.43 cm between stereovision acquisitions exceeds registration accuracy of 2mm. Lateral bending of the spine contributes most to this effect compared to axial rotation and flexion-extension. In conclusion, the viability of using stereovision-to-stereovision registration to account for interoperative motion of the spine is shown through this simulation. It is suggested the distance of spine movement between corresponding points does not surpass 2.43 cm between stereovision acquisitions.
Vascular navigation is an essential component of transcatheter cardiovascular interventions, conventionally performed using either 2D fluoroscopic imaging or CT- derived vascular roadmaps which can lead to many complications for the patients as well as the clinicians. This study presents an open-source and user-friendly 3D Slicer module that performs vessel reconstruction from tracked intracardiac ultrasound (ICE) imaging using deep learning-based methods. We also validate the methods by performing a vessel-phantom study. The results indicate that our Slicer module is able to reconstruct vessels with sufficient accuracy with an average distance error of 0.86 mm. Future work involves improving the speed of the methods as well as testing the module in an in-vivo setting. Clinical adaptation of this platform will allow the clinicians to navigate the vessels in 3D and will potentially enhance their spatial awareness as well as improve procedural safety.
Accurate lung nodule localization during Video-Assisted Thoracic Surgery (VATS) for the treatment of early-stage lung cancer is a surgical challenge. Recently, a new minimally invasive approach for nodule localization during VATS has been proposed, which consists in compensating by a biomechanical model the very large lung deformations occurring before and during surgery. This estimation of the deformations allows to transfer the position of the nodule visible on the preoperative CT to an acquisition of the lung performed during the operation using a Cone-Beam CT scanner (CBCT two). But, in this approach, an additional CBCT acquisition (CBCT one) must also be acquired just after the patient is placed in the operative position in order to estimate the deformations due to the change of the patient’s position, from supine during the CT acquisition to lateral decubitus in the operating room. Our goal is to simplify this procedure and thus reduce the radiation dose to the patient. To this end, we propose to improve this solution by replacing the CBCT one acquisition by a model allowing to predict these deformations. This model is defined using the lung state information from CBCT two and a general statistical motion model built from the position change deformations already observed on other patients. We have data from 17 patients. The method is evaluated with a leave-one-out cross-validation on its ability to reproduce the observed deformations. The method reduces the average prediction error from 12.12 mm without prediction to 8.09 mm for an average prediction, and finally to 6.33 mm for a prediction with our model fitted to CBCT two only.
Two main features of near-infrared (NIR) light are the ability to perform component analysis based on spectral differences and to have permeability to biological tissues. These features make the technology to acquire NIR spectral of the deep lesion and analyze the components by each pixel, called hyperspectral imaging (HSI). Mounting this technology to a laparoscope enables visualization of invisible or looking-similar tissues in visible light during laparoscopic surgery. In this research, the developed NIR-HSI laparoscopic device acquired NIR spectrum images on in vivo pig specimens. Through the experiments, the difference in spectrum between the artery and surrounding other tissues was confirmed. Additionally, a machine learning procedure provided high accuracy detection of the artery area; accuracy, precision, and recall are 0.868 %, 0.921 %, and 0.637 % respectively.
In this paper, we present a segmentation method for laparoscopic images using semantically similar groups for multi-class semantic segmentation. Accurate semantic segmentation is a key problem for computer assisted surgeries. Common segmentation models do not explicitly learn similarities between classes. We propose a model that, in addition to learning to segment an image into classes, also learns to segment it into human-defined semantically similar groups. We modify the LinkNet34 architecture by adding a second decoder with an auxiliary task of segmenting the image into these groups. The feature maps of the second decoder are merged into the final decoder. We validate our method against our base model LinkNet34 and a larger LinkNet50. We find that our proposed modification increased the performance both with mean Dice (average +1.5%) and mean Intersection over Union metrics (average +2.8%) on two laparoscopic datasets.
Breast cancer is a leading cause of death among women in the United States, and Breast Conserving Surgery (BCS) is a common treatment option for women with early-stage breast cancer. The goal of BCS is to localize and remove the tumor with negative margins, and incomplete tumor excision can lead to reoperation and recurrence. To aid with intraoperative tumor localization, a BCS image guidance system that features an optical tracking camera (NDI Polaris Vicra) and stereocameras (FLIR Grasshopper stereocamera pair) for tracking surgical tools and surgical scene monitoring, respectively, is being investigated. However, sensor data collected from both stereovision systems are not inherently co-registered. Thus, this paper proposes utilizing a tracked checkerboard calibration object paired with a custom 3DSlicer module for optical tracker and stereo camera co-registration. The module features an easy-to-use user interface to streamline calibration. The calibration process was validated with a tissue-mimicking breast phantom embedded with beads to represent surface fiducials and undergoing deformations (n=5 mock phantom tissue states). Across five independent calibration trials, the average Fiducial Registration Error (FRE) was calculated to be 0.68mm ± 0.11mm while the average Target Registration Error (TRE) was calculated to be 0.35mm ± 0.14mm. With respect to the challenge test conditions using the mock breast phantom under differing deformations, TRE values were on average determined to be 2.58mm ± 0.26mm over all breast phantom states. Thus, a tracked checkerboard tool paired with the custom 3DSlicer module provided the ability to localize and track points of interest for accurate registration of two different optical tracking systems.
Estimation of surgical tool pose is essential for surgical image guidance. Near real-time position and angle estimation is crucial, for example, for intraoperative optical coherence tomography tracking in retinal microsurgery. The current state-of-the-art algorithm for surgical tool tracking in posterior eye surgery was first introduced by Alsheakhali et al.1 We propose Dual Color Space Algorithm - an improved tool segmentation method based on combined color space masks, set thresholds, a shadow-insensitive detector for the tool edge, and more robust detection of the tip of the surgical tools. The presented algorithms are benchmarked on a series of manually annotated images from posterior eye surgery video. The video frames suffer from the confounding effect of the tool’s shadow and spot illumination occurring several times. A severalfold improvement in the algorithm’s accuracy is reported.
This paper advances a new paradigm of minimally invasive neurosurgical interventions through skull foramina, which promise to improve patient outcomes by reducing postoperative pain and recovery times, and perhaps even complication rates. The foramen ovale, a small opening in the base of the skull, is currently used to insert recording electrodes into the brain for diagnosing epilepsy and as a pathway for ablating the trigeminal nerve for facial pain. An MRI-compatible robotic platform to position neurosurgical tools along a prescribed trajectory through the foramen ovale can enable access to deep brain targets for diagnosis or intervention. In this paper, we describe design goals and constraints, determined both heuristically and empirically, for such a robotic system. These include the space available within the scanner around the patient, the set of possible needle angles of approach to the foramen ovale, patient positioning options within the scanner, and the force needed to tilt the needle to desired angles. These design considerations can be used to inform future work on the design of MRI-conditional robots to access the brain through the foramen ovale.
