Peripheral nerve block treatments are ubiquitous in hospitals and pain clinics worldwide. State of the art techniques use ultrasound (US) guidance and/or electrical stimulation to verify needle tip location. However, problems such as needle-US beam alignment, poor echogenicity of block needles and US beam thickness can make it difficult for the anesthetist to know the exact needle tip location. Inaccurate therapy delivery raises obvious safety and efficacy issues. We have developed and evaluated a needle guidance system that makes use of a magnetic tracking system (MTS) to provide an augmented reality (AR) guidance platform to accurately localize the needle tip as well as its projected trajectory. Five anesthetists and five novices performed simulated nerve block deliveries in a polyvinyl alcohol phantom to compare needle guidance under US alone to US placed in our AR environment. Our phantom study demonstrated a decrease in targeting attempts, decrease in contacting of critical structures, and an increase in accuracy of 0.68 mm compared to 1.34mm RMS in US guidance alone. Currently, the MTS uses 18 and 21 gauge hypodermic needles with a 5 degree of freedom sensor located at the needle tip. These needles can only be sterilized using an ethylene oxide process. In the interest of providing clinicians with a simple and efficient guidance system, we also evaluated attaching the sensor at the needle hub as a simple clip-on device. To do this, we simultaneously performed a needle bending study to assess the reliability of a hub-based sensor.

Trans-esophageal echocardiography (TEE) is a standard component of patient monitoring during most cardiac surgeries. In recent years magnetic tracking systems (MTS) have become sufficiently robust to function effectively in appropriately structured operating room environments. The ability to track a conventional multiplanar 2D TEE transducer in 3D space offers incredible potential by greatly expanding the cumulative field of view of cardiac anatomy beyond the limited field of view provided by 2D and 3D TEE technology. However, there is currently no TEE probe manufactured with MTS technology embedded in the transducer, which means sensors must be attached to the outer surface of the TEE. This leads to potential safety issues for patients, as well as potential damage to the sensor during procedures. This paper presents a standard 2D TEE probe fully integrated with MTS technology. The system is evaluated in an environment free of magnetic and electromagnetic disturbances, as well as a clinical operating room in the presence of a da Vinci robotic system. Our first integrated TEE device is currently being used in animal studies for virtual reality-enhanced ultrasound guidance of intracardiac surgeries, while the "second generation" TEE is in use in a clinical operating room as part of a project to measure perioperative heart shift and optimal port placement for robotic cardiac surgery. We demonstrate excellent system accuracy for both applications.

In recent years, magnetic tracking systems, whose fundamental unit of measurement is a 5D transformation (three translational and two rotational degrees-of-freedom), have become much more popular. Two 5D sensors can be combined to obtain a 6D transformation similar to the ones provided by the point-based registration in optical tracking. However, estimates of the tool tip uncertainty, which we have called the target tracking error (TTE) since no registration is explicitly performed, are not available in the same manner as their optical counterpart. If the systematic bias error can be corrected and estimates of the 5D or 6D fiducial localizer error (FLE) are provided in the form of zero mean normally distributed random variables in ¿ 5 and ¿ 6 , respectively, then the TTE can be modeled. In this paper, the required expressions that model the TTE as a function of the systematic bias, FLE and target location are derived and then validated using Monte Carlo simulations. We also show that the first order approximation is sufficient beyond the range of errors typically observed during an image-guided surgery (IGS) procedure. Applications of the models are described for a minimally invasive intracardiac surgical guidance system and needle-based therapy systems. Together with the target registration error (TRE) statistical models for point-based registration, the models presented in this article provide the basic framework for estimating the total system measurement uncertainty for an IGS system. Future work includes developing TRE models for commonly used registration methods that do not already have them.

Image-guided surgery systems have a wide range of applications where the level of accuracy required for each application varies from millimeters to low sub-millimeter range. In systems that use optical tracking, it is typical to use point-based registration without any weighting schemes to determine the pose of the tracked tool with very good accuracy. However, recent advancements in methods to estimate the measurement uncertainty for each tracked marker and the development of an anisotropically weighted point-based registration algorithm have allowed for the optical tracking accuracy to be improved. In this article, we demonstrate a new tracking method that improves the tracking accuracy by 20-45% over the traditional tracking methodology.

Anesthetic nerve blocks are a common therapy performed in hospitals around the world to alleviate acute and chronic pain. Tracking systems have shown considerable promise in other forms of therapy, but little has been done to apply this technology in the field of anesthesia. We are developing a guidance system for combining tracked needles with non-invasive ultrasound (US) and patient-specific geometric models. In experiments with phantoms two augmented reality (AR) guidance systems were compared to the exclusive use of US for lumbar facet injection therapy. Anesthetists and anesthesia residents were able to place needles within 0.57mm of the intended targets using our AR systems compared to 5.77 mm using US alone. A preliminary cadaver study demonstrated the system was able to accurately place radio opaque dye on targets. The combination of real time US with tracked tools and AR guidance has the potential to replace CT and fluoroscopic guidance, thus reducing radiation dose to patients and clinicians, as well as reducing health care costs.

Ultrasound is garnering significant interest as an imaging modality for surgical guidance, due to its affordability, real-time temporal resolution and ease of integration into the operating room. Minimally-invasive intracardiac surgery performed on the beating-heart prevents direct vision of the surgical target, and procedures such as mitral valve replacement and atrial septal defect closure would benefit from intraoperative ultrasound imaging. We propose that placing 4D ultrasound within an augmented reality environment, along with a patient-specific cardiac model and virtual representations of tracked surgical tools, will create a visually intuitive platform with sufficient image information to safely and accurately repair tissue within the beating heart. However, the quality of the imaging parameters, spatial calibration, temporal calibration and ECG-gating must be well characterized before any 4D ultrasound system can be used clinically to guide the treatment of moving structures. In this paper, we describe a comprehensive accuracy assessment framework that can be used to evaluate the performance of 4D ultrasound systems while imaging moving targets. We image a dynamic phantom that is comprised of a simple robot and a tracked phantom to which point-source, distance and spherical objects of known construction can be attached. We also follow our protocol to evaluate 4D ultrasound images generated in real-time by reconstructing ECG-gated 2D ultrasound images acquired from a tracked multiplanar transesophageal probe. Likewise, our evaluation framework allows any type of 4D ultrasound to be quantitatively assessed.

In image-guide surgery, optimizing the accuracy in localizing the surgical tools within the virtual reality environment or 3D image is vitally important, significant effort has been spent reducing the measurement errors at the point of interest or target. This target registration error (TRE) is often defined by a root-mean-square statistic which reduces the vector data to a single term that can be minimized. However, lost in the data reduction is the directionality of the error which, can be modelled using a 3D covariance matrix. Recently, we developed a set of expressions that modeled the TRE statistics for point-based registrations as a function of the fiducial marker geometry, target location and the fiducial localizer error (FLE). Unfortunately, these expressions are only as good as the definition of the FLE. In order to close the gap, we have subsequently developed a closed form expression that estimates the FLE as a function of the estimated fiducial registration error (FRE, the error between the measured fiducials and the best fit locations of those fiducials). The FRE covariance matrix is estimated using a sliding window technique and used as input into the closed form expression to estimate the FLE. The estimated FLE can then used to estimate the TRE which, can be given to the surgeon to permit the procedure to be designed such that the errors associated with the point-based registrations are minimized.

Target registration error (TRE) has become a widely accepted error metric in point-based registration since the error metric was introduced in the 1990s. It is particularly prominent in image-guided surgery (IGS) applications where point-based registration is used in both image registration and optical tracking. In point-based registration, the TRE is a function of the fiducial marker geometry, location of the target and the fiducial localizer error (FLE). While the first two items are easily obtained, the FLE is usually estimated using an a priori technique and applied without any knowledge of real-time information. However, if the FLE can be estimated in real-time, particularly as it pertains to optical tracking, then the TRE can be estimated more robustly. In this paper, a method is presented where the FLE statistics are estimated from the latest measurement of the fiducial registration error (FRE) statistics. The solution is obtained by solving a linear system of equations of the form Ax=b for each marker at each time frame where x are the six independent FLE covariance parameters and b are the six independent estimated FRE covariance parameters. The A matrix is only a function of the tool geometry and hence the inverse of the matrix can be computed a priori and used at each instant in which the FLE estimation is required, hence minimizing the level of computation at each frame. When using a good estimate of the FRE statistics, Monte Carlo simulations demonstrate that the root mean square of the FLE can be computed within a range of 70-90 mum. Robust estimation of the TRE for an optically tracked tool, using a good estimate of the FLE, will provide two enhancements in IGS. First, better patient to image registration will be obtained by using the TRE of the optical tool as a weighting factor of point-based registration used to map the patient to image space. Second, the directionality of the TRE can be relayed back to the surgeon giving the surgeon the option of changing their strategy in order to improve the overall system accuracy and, in turn, the quality of procedure.

In image-guided interventions, anatomical models of organs are often generated from pre-operative images and further employed in planning and guiding therapeutic procedures. However, the accuracy of these models, along with their registration to the subject are crucial for successful therapy delivery. These factors are amplified when manipulating soft tissue undergoing large deformations, such as the heart. When used in guiding beating-heart procedures, pre-operative models may not be sufficient for guidance and they are often complemented with real-time, intra-operative cardiac imaging. Here we demonstrate via in vitro endocardial "therapy" that ultrasound-enhanced model-guided navigation provides sufficient guidance to preserve a clinically-desired targeting accuracy of under 3 mm independently of the model-to-subject misregistrations. These results emphasize the direct benefit of integrating real-time imaging within intra-operative visualization environments considering that model-to-subject misalignments are often encountered clinically.

Catheter-driven cardiac interventions have emerged in response to the need of reducing invasiveness associated with the traditional cut-and-sew techniques. Catheter manipulation is traditionally performed under real-time fluoroscopy imaging, resulting in an overall trade-off of procedure invasiveness for radiation exposure of both the patient and clinical staff. Our approach to reducing and potentially eliminating the use of flouroscopy in the operating room entails the use of multi-modality imaging and magnetic tracking technologies, wrapped together into an augmented reality environment for enhanced intra-procedure visualization and guidance. Here we performed an in vitro study in which a catheter was guided to specific targets located on the endocardial atrial surface of a beating heart phantom. "Therapy delivery" was modeled in the context of a blinded procedure, mimicking a beating heart, intracardiac intervention. The users navigated the tip of a magnetically tracked Freezor 5 CRYOCATH catheter to the specified targets. Procedure accuracy was determined as the distance between the tracked catheter tip and the tracked surgical target at the time of contact, and it was assessed under three different guidance modalities: endoscopic, augmented reality, and ultrasound image guidance. The overall RMS targeting accuracy achieved under augmented reality guidance averaged to 1.1 mm. This guidance modality shows significant improvements in both procedure accuracy and duration over ultrasound image guidance alone, while maintianing an overall targeting accuracy comparable to that achieved under endoscopic guidance.

Minimally invasive beating heart intracardiac surgery is an area of research with many unique challenges. Surgical targets are in constant motion in a blood-filled environment that prevents direct line-of-sight guidance. The restrictive workspace requires compact, yet robust tools for proper therapy delivery. Our novel method for approaching multiple targets inside the beating heart allows their identification and access under augmented reality-assisted image guidance. The surgical platform integrates real-time ultrasound imaging with virtual models of the surgical instruments, along with virtual cardiac anatomy acquired from pre-operative images. Extensive in vitro studies were performed to assess the operator's ability to "deliver therapy" to dynamic intracardiac targets via both transmural and transluminal access, and demonstrated significantly more accurate targeting under augmented reality guidance compared to ultrasound image guidance alone, accompanied by a reduction of procedure time by half. Moreover, preliminary in vivo acute studies on porcine models showed successful prosthesis positioning for beating-heart septal defect repair and mitral valve implantation via direct surgical access. While still in its infancy, this work emphasizes the promise of ultrasound-enhanced model-guided environments for minimally-invasive cardiac therapy, whether delivered via a catheter introduced into the vascular system or a cannula inserted through the heart wall.

A 2D ultrasound enhanced virtual reality surgical guidance system has been under development for some time in our lab. The new surgical guidance platform has been shown to be effective in both the laboratory and clinical settings, however, the accuracy of the tracked 2D ultrasound has not been investigated in detail in terms of the applications for which we intend to use it (i.e., mitral valve replacement and atrial septal defect closure). This work focuses on the development of an accuracy assessment protocol specific to the assessment of the calibration methods used to determine the rigid transformation between the ultrasound image and the tracked sensor. Specifically, we test a Z-bar phantom calibration method and a phantomless calibration method and compared the accuracy of tracking ultrasound images from neuro, transesophageal, intracardiac and laparoscopic ultrasound transducers. This work provides a fundamental quantitative description of the image-guided accuracy that can be obtained with this new surgical guidance system.

Error models associated with point-based medical image registration problems were first introduced in the late 1990s. The concepts of fiducial localizer error, fiducial registration error, and target registration error are commonly used in the literature. The model for estimating the target registration error at a position r in a coordinate frame defined by a set of fiducial markers rigidly fixed relative to one another is ubiquitous in the medical imaging literature. The model has also been extended to simulate the target registration error at the point of interest in optically tracked tools. However, the model is limited to describing the error in situations where the fiducial localizer error is assumed to have an isotropic normal distribution in R3. In this work, the model is generalized to include a fiducial localizer error that has an anisotropic normal distribution. Similar to the previous models, the root mean square statistic rmstre is provided along with an extension that provides the covariance matrix Sigmatre. The new model is verified using a Monte Carlo simulation and a set of statistical hypothesis tests. Finally, the differences between the two assumptions, isotropic and anisotropic, are discussed within the context of their use in 1) optical tool tracking simulation and 2) image registration.

As part of an ongoing theme in our laboratory on reducing morbidity during minimally-invasive intracardiac procedures, we developed a computer-assisted intervention system that provides safe access inside the beating heart and sufficient visualization to deliver therapy to intracardiac targets while maintaining the efficacy of the procedure. Integrating pre-operative information, 2D trans-esophageal ultrasound for real-time intra-operative imaging, and surgical tool tracking using the NDI Aurora magnetic tracking system in an augmented virtual environment, our system allows the surgeons to navigate instruments inside the heart in spite of the lack of direct target visualization. This work focuses on further enhancing intracardiac visualization and navigation by supplying the surgeons with detailed 3D dynamic cardiac models constructed from high-resolution pre-operative MR data and overlaid onto the intra-operative imaging environment. Here we report our experience during an in vivo porcine study. A feature-based registration technique previously explored and validated in our laboratory was employed for the pre-operative to intra-operative mapping. This registration method is suitable for in vivo interventional applications as it involves the selection of easily identifiable landmarks, while ensuring a good alignment of the pre-operative and intra-operative surgical targets. The resulting augmented reality environment fuses the pre-operative cardiac model with the intra-operative real-time US images with approximately 5 mm accuracy for structures located in the vicinity of the valvular region. Therefore, we strongly believe that our augmented virtual environment significantly enhances intracardiac navigation of surgical instruments, while on-target detailed manipulations are performed under real-time US guidance.

In the context of our ongoing objective to reduce morbidity associated with cardiac interventions, minimizing invasiveness has inevitably led to more limited visual access to the target tissues. To ameliorate these challenges, we provide the surgeons with a complex visualization environment that integrates interventional ultrasound imaging augmented with pre-operative anatomical models and virtual surgical instruments within a virtual reality environment. In this paper we present an in vitro study on a cardiac phantom aimed at assessing the feasibility and targeting accuracy of our surgical system in comparison to traditional ultrasound imaging for intra-operative surgical guidance. The “therapy delivery” was modeled in the context of a blinded procedure, mimicking a closed-chest intervention. Four users navigated a tracked pointer to a target, under guidance provide by either US imaging or virtual reality-enhanced ultrasound. A 2.8 mm RMS targeting error was achieved using our novel surgical system, which is adequate from both a clinical and engineering perspective, under the inherent procedure requirements and limitations of the system.

Cardiopulmonary bypass surgery, although a highly invasive interventional approach leading to numerous complications, is still the most common therapy option for treating many forms of cardiac disease. We are currently engaged in a project designed to replace many bypass surgeries with less traumatic, minimally invasive intracardiac therapies. This project combines real-time intra-operative echocardiography with a virtual reality environment providing the surgeon with a broad range of valuable information. Pre-operative images, electrophysiological data, positions of magnetically tracked surgical instruments, and dynamic surgical target representations are among the data that can be presented to the surgeon to augment intra-operative ultrasound images. This augmented reality system is applicable to procedures such as mitral valve replacement and atrial septal defect repair, as well as ablation therapies for treatment of atrial fibrillation. Our goal is to develop a robust augmented reality system that will improve the efficacy of intracardiac treatments and broaden the range of cardiac surgeries that can be performed in a minimally invasive manner. This paper provides an overview of our interventional system and specific experiments that assess its pre-clinical performance.

A 2D ultrasound enhanced virtual reality surgical guidance system has been developed in our laboratory. The system was tested in both the laboratory and the clinic. Recently, we studied the accuracy of two the ultrasound calibration methods with five different ultrasound transducers using a spherical object as the test platform. In this paper, we extend that work to use the superior vena cava and right atrium of a beating heart phantom as the metrological test artefact. The right atrium was imaged using the tracked ultrasound and the expected cross-sectional outline was determined using the intersection of the ultrasound plane and the surface model in the surgical guidance system. The expected and observed outlines are compared. The results show that the ultrasound calibration methods were sufficiently accurate in the spatial domain, but that temporal calibration is required to ensure accuracy throughout a given procedure.

In an effort to reduce morbidity of cardiac interventions, minimizing invasiveness inevitably leads to limited visual access to the surgical targets. To address these limitations, we provide the surgeons with a robust visualization environment that integrates interventional ultrasound imaging augmented with pre-operative anatomical models and virtual surgical instruments within a virtual reality environment. Here we present an in vitro study on a cardiac phantom that mimics an ablation therapy procedure, which allows us to assess the feasibility of our surgical system in comparison to traditional intra-operative ultrasound imaging. Following surgical target identification via an electro-anatomical model, the "ablation procedure" is performed blindly. A 2.8 mm RMS targeting error is achieved using our novel surgical system. This level of accuracy is adequate from both a clinical and engineering perspective, under the inherent procedure requirements and limitations of the system.

Many intracardiac procedures can currently be performed on the heart only after it has been arrested, and the patient has been placed on cardio-pulmonary bypass. We have developed a new method for operating on multiple targets inside the beating heart, and describe a procedure for accessing them under virtual-reality (VR)-assisted image guidance that combines real-time ultrasound with a virtual model of tools, and the surgical environment acquired from pre-operative images. This paper presents preliminary results aimed at assessing the operator's ability to accurately position and staple an artificial valve to a "valve orifice" within a cardiac phantom when guidance is performed via ultrasound alone, and with US augmented by the VR environment.