Locating standard scan planes in ultrasound (US)-guided procedures is operator-dependent and time-consuming, often requiring experienced sonographers. We propose a deep reinforcement learning framework for autonomous probe guidance to localize the long-axis (dorsal) kidney scan plane. A dueling deep Q-network was trained in silico using US images synthesized from dynamically resliced CT volumes, with an image-based reward derived from CT-labeled anatomy. On unseen simulated data, the agent achieved an 80.4% success rate, reaching the target in under 34 seconds. The image-based reward improved the success rate by ~50% compared to a distance-based reward, with the largest gains in higher-degree-of-freedom (5-DoF) tasks. Real-world validation showed 75.8% success on a phantom model and 74.0% on a cadaver specimen. These results demonstrate strong sim-to-real generalization and the feasibility of autonomous scan plane localization for renal imaging, with potential applicability to broader US-guided interventional procedures and robotic imaging workflows.

Purpose. Magnetic resonance (MR) imaging offers high-quality soft tissue visualization for diagnosing spinal conditions; however, its use in surgical navigation is challenged by anatomical deformations resulting from differences in patient positioning. This study introduces a multi-stage, multi-resolution approach for automated initialization of registration between preoperative 3D and intraoperative Long-Film (LF) images, thus eliminating the need for manual segmentation and labeling of individual vertebrae. Methods. A convolutional neural network was designed and implemented to regress the 3D poses of vertebrae using features extracted from soft-tissue-suppressed 2D images. The approach operates within a multi-resolution pyramid to initialize registrations from global to local scale. To improve robustness against large misalignments and outliers, a batched multi-start strategy was used. The network was trained using simulated LF images from 365 CT and evaluated on real CT and LF images from a cadaver study of 14 cases. Results. The proposed multi- stage initialization and batched inference approach achieved a median target registration error (TRE) of 6.6 mm (interquartile range: 3.9-10.2 mm) in cadaver studies, demonstrating accuracy comparable to the initial TRE range reported in previous studies that utilized model-based optimization techniques. Conclusions. This study reports a multi-stage initialization method for 3D-2D pose regression. Initial results demonstrate the feasibility of using pose regression networks for initial alignment of preoperative 3D and intraoperative LF images. Ongoing work aims to extend this approach to a fully automated, end-to-end multi-modal registration pipeline.

Objective: Transventricular approach to deep-brain targets offers direct visualization but also imparts deformation that challenges accurate neuronavigation. 3D reconstruction and registration of the endoscopic view could provide up-to-date, realtime guidance. We develop and evaluate a self-supervised feature detection method for 3D reconstruction and navigation in neuroendoscopy. Methods: Unlabeled neuroendoscopic video data from 15 clinical cases yielding 11,527 video frames yielding 11,527 video frames were used to train a self-supervised learning method (R2D2-E) with 5-fold cross validation integrated into a simultaneous localization and mapping (SLAM) pipeline for 3D reconstruction. A series of experiments guided nominal hyperparameters selection and evaluated performance in comparison to SIFT, SURF and SuperPoint in terms of the accuracy of feature matching and 3D reconstruction. Results: R2D2-E demonstrated a superior performance in feature matching and 3D reconstruction. R2D2-E features achieved a median projected error of 0.64 mm compared to 0.90 mm, 0.99 mm and 0.83 mm error for SIFT, SURF and SuperPoint, respectively. The method also improved F1 score by 14%, 25% and 22% compared to SIFT, SURF and SuperPoint, respectively. Conclusion: The proposed feature detection approach enables accurate, real-time 3D reconstruction in neuroendoscopy, offering robust feature detection in the presence of endoscopic artifacts and provides up-to-date navigation following soft-tissue deformation. Significance: The self-supervised feature detection method advances capabilities for vision-based guidance and augmented visualization of target structures in neuroendoscopic procedures. The approach could enhance the accuracy and precision of neurosurgery to improve patient outcomes.

Purpose. Locating appropriate scan planes in ultrasound (US) imaging is a critical first task that can be time-consuming and highly user-dependent. The choice between freehand and robotic techniques depends on the specific use case, the environment in which the ultrasound is being performed, and the priorities of the healthcare provider. This work aims to compare and evaluate these two techniques for the task of localizing standard scan planes in renal imaging. Methods. A prototype robotic US imaging system, integrated with a reinforcement learning (RL) agent, was developed and tasked with localizing the longitudinal scan plane of the kidney. Simulated perturbations were introduced to emulate potential errors in freehand acquisition, such as deviations in following directions and maintaining contact forces. The performance of the guidance system was evaluated on simulated images, a phantom model, and a cadaveric specimen in terms of the agent's accuracy and efficiency in reaching the target. Results. The proposed method performed comparably in both freehand and robotic approaches. The freehand approach achieved similar levels of reachability (89 +/- 2%) as the robotic approach, despite inconsistencies in following the agent's directions. The robotic approach had better overall efficiency (40 degrees total travel) in reaching the target (i.e., taking shorter paths to reach the target plane). Acquired images had better consistency, as demonstrated by the higher-quality 3D volume reconstructions. The findings were successfully replicated in phantom and cadaver specimens. Conclusions. This work reports a comparison between freehand and robotic methods for the task of localizing standard scan planes. The integrated system enabled emulation of freehand and robotic operations, which were systematically compared in manipulation errors, force control deviations, and validated in phantom and cadaver studies. The comparable performance of the freehand approach suggests its suitability in applications, such as point-of-care imaging, where robotic US may not be feasible.

Purpose. Finding desired scan planes in ultrasound (US) imaging is a critical first task that can be time-consuming, influenced by operator experience, and subject to inter-operator variability. To circumvent these problems, interventional US imaging often necessitates dedicated, experienced sonographers in the operating room. This work presents a new approach leveraging deep reinforcement learning (RL) to assist probe positioning. Methods. A deep Q-network (DQN) is applied and evaluated for renal imaging and is tasked with locating the dorsal US scan plane. To circumvent the need for large labeled datasets, images were resliced from a large dataset of CT volumes and synthesized to US images using Field II, CycleGAN, and U-GAT-IT. The algorithm was evaluated on both synthesized and real US images, and its performance was quantified in terms of the agent's accuracy in reaching the target scan plane. Results. Learning-based synthesis methods performed better than the physics-based approach, achieving comparable image quality when qualitatively compared to real US images. The RL agent was successful in reaching target scan planes when adjusting the probe's rotation, with the U-GAT-IT model demonstrating superior generalizability (80.3% reachability) compared to CycleGAN (54.8% reachability). Conclusions. The approach presents a novel RL training strategy using image synthesis for automated US probe positioning. Ongoing efforts aim to evaluate advanced DQN models, image-based reward functions, and support probe motion with higher degrees of freedom.

Purpose: Navigating deep-brain structures in neurosurgery, especially under deformation from CSF egress, remains challenging due to the limitations of current robotic systems relying on rigid registration. This study presents the initial steps towards vision-based navigation leveraging Neural Radiance Fields (NeRF) to enable 3D neuroendoscopic reconstruction on the Robot-Assisted Ventriculoscopy (RAV) platform. Methods: An end-to-end 3D reconstruction and registration method using posed images was developed and integrated with the RAV platform. The hyperparameters for training the dual-branch network were first identified. Further experiments were conducted to evaluate reconstruction accuracy using projected error (PE) while varying the volume density threshold parameter. Results: A 3D volume was reconstructed using a simple linear trajectory for data acquisition with 300 frames and corresponding camera poses. The density volume threshold was varied to obtain an optimal value of 96.55 percentile, with a corresponding PE of 0.65 mm. Conclusions: Initial methods for end-to-end neuroendoscopic video reconstruction were developed in phantom studies. Experiments identified the optimal parameters, yielding a geometrically accurate reconstruction along with fast network convergence runtime of < 30 s. The method is highly promising for future clinical translation in realistic neuroendoscopic scenes. Future work will also develop a direct surface-to-volume registration method for improving reconstruction accuracy and runtime.

PurposeImproved integration and use of preoperative imaging during surgery hold significant potential for enhancing treatment planning and instrument guidance through surgical navigation. Despite its prevalent use in diagnostic settings, MR imaging is rarely used for navigation in spine surgery. This study aims to leverage MR imaging for intraoperative visualization of spine anatomy, particularly in cases where CT imaging is unavailable or when minimizing radiation exposure is essential, such as in pediatric surgery.MethodsThis work presents a method for deformable 3D-2D registration of preoperative MR images with a novel intraoperative long-length tomosynthesis imaging modality (viz., Long-Film [LF]). A conditional generative adversarial network is used to translate MR images to an intermediate bone image suitable for registration, followed by a model-based 3D-2D registration algorithm to deformably map the synthesized images to LF images. The algorithm’s performance was evaluated on cadaveric specimens with implanted markers and controlled deformation, and in clinical images of patients undergoing spine surgery as part of a large-scale clinical study on LF imaging.ResultsThe proposed method yielded a median 2D projection distance error of 2.0mm (interquartile range [IQR]: 1.1-3.3mm) and a 3D target registration error of 1.5mm (IQR: 0.8-2.1mm) in cadaver studies. Notably, the multi-scale approach exhibited significantly higher accuracy compared to rigid solutions and effectively managed the challenges posed by piecewise rigid spine deformation. The robustness and consistency of the method were evaluated on clinical images, yielding no outliers on vertebrae without surgical instrumentation and 3% outliers on vertebrae with instrumentation.ConclusionsThis work constitutes the first reported approach for deformable MR to LF registration based on deep image synthesis. The proposed framework provides access to the preoperative annotations and planning information during surgery and enables surgical navigation within the context of MR images and/or dual-plane LF images.

Purpose: Deep Learning (DL) image synthesis has gained increasing popularity for the reconstruction of CT and cone-beam CT (CBCT) images, especially in combination with physically-principled reconstruction algorithms. However, DL synthesis is challenged by the generalizability of training data and noise in the trained model. Epistemic uncertainty has proven as an efficient way of quantifying erroneous synthesis in the presence of out-of-domain features, but its estimation with Monte Carlo (MC) dropout requires a large number of inference runs, variable as a function of the particular uncertain feature. We propose a single-pass method - the Moment Propagation Model - which approximates the MC dropout by analytically propagating the statistical moments through the network layers, removing the need for multiple inferences and removing errors in estimations from insufficient dropout realizations. Methods: The proposed approach jointly computes the change of the expectation and the variance of the input (first two statistical moments) through each network layer, where each moment undergoes a different numerical transformation. The expectation is initialized as the network input; the variance is solely introduced at dropout layers, modeled as a Bernoulli process. The method was evaluated using a 3D Bayesian conditional generative adversarial network (GAN) for synthesis of high-quality head MDCT from low-quality intraoperative CBCT reconstructions. 20 pairs of measured MDCT volumes (120 kV, 400-550 mAs) depicting normal head anatomy, and simulated CBCT volumes (100-120 kV, 32-200 mAs) were used for training. Scatter, beam-hardening, detector lag and glare were added to the simulated CBCT and were corrected (assuming unknown) prior to reconstruction. Epistemic uncertainty was estimated for 30 heads (outside of the training set) containing simulated brain lesions using the proposed single-pass propagation model, and results were compared to the standard 200-pass dropout approach. Image quality and quantitative accuracy of the estimated uncertainty of lesions and other anatomical sites were further evaluated. Results: The proposed propagation model captured >2 HU increase in epistemic uncertainty caused by various hyper- and hypo-density lesions, with <0.31 HU error over the brain compared to the reference MC dropout result at 200 inferences and <0.1 HU difference to a converged MC dropout estimate at 100 inference passes. These findings indicate a potential 100-fold increase in computational efficiency of neural network uncertainty estimation. Conclusions: The proposed moment propagation model is able to achieve accurate quantification of epistemic uncertainty in a single network pass and is an efficient alternative to conventional MC dropout.

A new imaging modality (viz., Long-Film [LF]) for acquiring long-length tomosynthesis images of the spine was recently enabled on the O-arm™ system and used in an IRB-approved clinical study at our institution. The work presented here implements and evaluates a combined image synthesis and registration approach to solve multi-modality registration of MR and LF images. The approach is well-suited for pediatric cases that use MR for preoperative diagnosis and aim for lower levels of intraoperative radiation exposure. A patch-based conditional GAN was used to synthesize 3D CT images from MR. The network was trained on deformably co-registered MR and CT image pairs. Synthesized images were registered to LF images using a model-based 3D-2D registration algorithm. Images from our clinical study were manually labeled, and the intra-user variability in anatomical landmark definition was measured in a simulation study. Geometric accuracy of registrations was evaluated on anatomical landmarks in separate test cases from the clinical study. The synthesis process generated CT images with clear bone structures. Analysis of manual labeling revealed 3.1±2.2 mm projection distance error between 3D and 2D anatomical landmarks. Anatomical MR landmarks projected on lateral LF images demonstrated a median projection distance error of 3.6 mm after registration. This work constitutes the first reported approach for MR to LF registration based on deep image synthesis. Preliminary results demonstrated the feasibility of globally rigid registration in aligning preoperative MR and intraoperative LF images. Work currently underway extends this approach to vertebra level, locally rigid / globally deformable registrations, with initialization based on automatically labeled vertebrae levels.

Neurosurgical techniques often require accurate targeting of deep-brain structures even in the presence of deformation due intervention and egress of Cerebrospinal Fluid (CSF) during surgical access. Prior work reported Simultaneous Localization and Mapping (SLAM) methods for endoscopic guidance using 3D reconstruction. In this work, methods for correcting the geometric distortion of a neuroendoscope are reported in a form that have been translated intraoperative use in first clinical studies. Furthermore, SLAM methods are evaluated in first clinical studies for real-time 3D endoscopic navigation with near real-time registration in the presence of deep-brain tissue deformation. A custom calibration jig with swivel mounts was designed and manufactured for neuroendoscope calibration in the operating room. The process is potentially suitable to intraoperative use while maintaining sterility of the endoscope, although the current calibration system was used in the Operating Room (OR) immediately following the case for offline analysis. A six by seven checkerboard pattern was used to obtain corner locations for calibration, and the method was evaluated in terms of Reprojection Error (RPE). Neuroendoscopic video was acquired under an IRB-approved clinical study, demonstrating rich vascular features and other structures on the interior walls of the lateral ventricles for 3D point-cloud reconstruction. Geometric accuracy was evaluated in terms of Projected Error (PE) on a ground truth surface defined from MR or cone-beam CT (CBCT) images. Intraoperative neuroendoscope calibration was achieved with sub-pixel [0.61 ± 0.20 px] error. The calibration yielded a focal length of 816.42 px and 822.71 px in X and Y directions respectively, along with radial distortion coefficients of -0.432 (first order term [𝑘1]) and 0.158 (second order term [𝑘2]). The 3D reconstruction was performed successfully with a PE of 0.23 ± 0.15 mm compared to the ground truth surface. The system for neuroendoscopic guidance based on SLAM 3D point-cloud reconstruction provided a promising platform for the development of 3D neuroendoscopy. The studies reported in this work presented an important means of neuroendoscope calibration in the OR and provided preliminary evidence for accurate 3D video reconstruction in first clinical studies. Future work aims to further extend the clinical evaluation and improve reconstruction accuracy using ventricular shape priors.

BACKGROUND:Image-guided neurosurgery requires high localization and registration accuracy to enable effective treatment and avoid complications. However, accurate neuronavigation based on preoperative magnetic resonance (MR) or computed tomography (CT) images is challenged by brain deformation occurring during the surgical intervention.PURPOSE:To facilitate intraoperative visualization of brain tissues and deformable registration with preoperative images, a 3D deep learning (DL) reconstruction framework (termed DL-Recon) was proposed for improved intraoperative cone-beam CT (CBCT) image quality.METHODS:The DL-Recon framework combines physics-based models with deep learning CT synthesis and leverages uncertainty information to promote robustness to unseen features. A 3D generative adversarial network (GAN) with a conditional loss function modulated by aleatoric uncertainty was developed for CBCT-to-CT synthesis. Epistemic uncertainty of the synthesis model was estimated via Monte Carlo (MC) dropout. Using spatially varying weights derived from epistemic uncertainty, the DL-Recon image combines the synthetic CT with an artifact-corrected filtered back-projection (FBP) reconstruction. In regions of high epistemic uncertainty, DL-Recon includes greater contribution from the FBP image. Twenty paired real CT and simulated CBCT images of the head were used for network training and validation, and experiments evaluated the performance of DL-Recon on CBCT images containing simulated and real brain lesions not present in the training data. Performance among learning- and physics-based methods was quantified in terms of structural similarity (SSIM) of the resulting image to diagnostic CT and Dice similarity metric (DSC) in lesion segmentation compared to ground truth. A pilot study was conducted involving seven subjects with CBCT images acquired during neurosurgery to assess the feasibility of DL-Recon in clinical data.RESULTS:CBCT images reconstructed via FBP with physics-based corrections exhibited the usual challenges to soft-tissue contrast resolution due to image non-uniformity, noise, and residual artifacts. GAN synthesis improved image uniformity and soft-tissue visibility but was subject to error in the shape and contrast of simulated lesions that were unseen in training. Incorporation of aleatoric uncertainty in synthesis loss improved estimation of epistemic uncertainty, with variable brain structures and unseen lesions exhibiting higher epistemic uncertainty. The DL-Recon approach mitigated synthesis errors while maintaining improvement in image quality, yielding 15%-22% increase in SSIM (image appearance compared to diagnostic CT) and up to 25% increase in DSC in lesion segmentation compared to FBP. Clear gains in visual image quality were also observed in real brain lesions and in clinical CBCT images.CONCLUSIONS:DL-Recon leveraged uncertainty estimation to combine the strengths of DL and physics-based reconstruction and demonstrated substantial improvements in the accuracy and quality of intraoperative CBCT. The improved soft-tissue contrast resolution could facilitate visualization of brain structures and support deformable registration with preoperative images, further extending the utility of intraoperative CBCT in image-guided neurosurgery.

Self-supervised learning has demonstrated state-of-the-art performance on various anomaly detection tasks. Learning effective representations by solving a supervised pretext task with pseudo-labels generated from unlabeled data provides a promising concept for industrial downstream tasks such as process monitoring. In this paper, we present SSMSPC a novel approach for multivariate statistical in-process control (MSPC) based on self-supervised learning. Our motivation for SSMSPC is to leverage the potential of unsupervised representation learning by incorporating self-supervised learning into the general statistical process control (SPC) framework to develop a holistic approach for the detection and localization of anomalous process behavior in discrete manufacturing processes. We propose a pretext task called Location + Transformation prediction, where the objective is to classify both, the type and the location of a randomly applied augmentation on a given time series input. In the downstream task, we follow the one-class classification setting and apply the Hotelling’s T^2 statistic on the learned representations. We further propose an extension to the control chart view that combines metadata with the learned representations to visualize the anomalous time steps in the process data which supports a machine operator in the root cause analysis. We evaluate the effectiveness of SSMSPC with two real-world CNC-milling datasets and show that it outperforms state-of-the-art anomaly detection approaches, achieving 100% and 99.6% AUROC, respectively. Lastly, we deploy SSMSPC at a CNC-milling machine to demonstrate its practical applicability when used as a process monitoring tool in a running process.

Neuroendoscopic approach to deep-brain targets imparts deformation of the ventricles and adjacent parenchyma, limiting the accuracy of conventional neuronavigation. We report a method for 3D endoscopic reconstruction and registration via simultaneous localization and mapping (SLAM) for real-time guidance with or without robotic assistance. The aim is to permit augmented video overlay of structures registered from preoperative or intraoperative 3D images within and beyond the endoscopic field of view for more accurate targeting in the presence of deep-brain deformation. Phantom studies were performed to evaluate geometric accuracy and uncertainty in distinct scenarios of limited data (feature sparsity and scene occlusion), demonstrating performance over a broad range of challenges to endoscopic data. Reconstruction and registration accuracy were maintained even with up to 40% loss in feature density or 120° of the visual scene occluded. Overall, the method achieved a high degree of geometric accuracy, with target registration error of 1.02 mm and runtime supporting real-time guidance (3.45 Hz, representing $\text{a}16\times $ speedup with SLAM approach compared to previous work). The studies establish essential quantitative performance characteristics and validation that are essential to future translation to clinical studies.

BACKGROUND:Indirect detection flat-panel detectors (FPDs) consisting of hydrogenated amorphous silicon (a-Si:H) thin-film transistors (TFTs) are a prevalent technology for digital x-ray imaging. However, their performance is challenged in applications requiring low exposure levels, high spatial resolution, and high frame rate. Emerging FPD designs using metal oxide TFTs may offer potential performance improvements compared to FPDs based on a-Si:H TFTs.PURPOSE:This work investigates the imaging performance of a new indium gallium zinc oxide (IGZO) TFT-based detector in 2D fluoroscopy and 3D cone-beam CT (CBCT).METHODS:The new FPD consists of a sensor array combining IGZO TFTs with a-Si:H photodiodes and a 0.7-mm thick CsI:Tl scintillator. The FPD was implemented on an x-ray imaging bench with system geometry emulating intraoperative CBCT. A conventional FPD with a-Si:H TFTs and a 0.6-mm thick CsI:Tl scintillator was similarly implemented as a basis of comparison. 2D imaging performance was characterized in terms of electronic noise, sensitivity, linearity, lag, spatial resolution (modulation transfer function, MTF), image noise (noise-power spectrum, NPS), and detective quantum efficiency (DQE) with entrance air kerma (EAK) ranging from 0.3 to 1.2 μGy. 3D imaging performance was evaluated in terms of the 3D MTF and noise-equivalent quanta (NEQ), soft-tissue contrast-to-noise ratio (CNR), and image quality evident in anthropomorphic phantoms for a range of anatomical sites and dose, with weighted air kerma, K w ${K_w}$ , ranging from 0.8 to 4.9 mGy.RESULTS:The 2D imaging performance of the IGZO-based FPD exhibited up to ∼1.7× lower electronic noise than the a-Si:H FPD at matched pixel pitch. Furthermore, the IGZO FPD exhibited ∼27% increase in mid-frequency DQE (1 mm-1 ) at matched pixel size and dose (EAK ≈ 1.0 μGy) and ∼11% increase after adjusting for differences in scintillator thickness. 2D spatial resolution was limited by the scintillator for each FPD. The IGZO-based FPD demonstrated improved 3D NEQ at all spatial frequencies in both head (≥25% increase for all dose levels) and body (≥10% increase for K w ${K_w}$ ≤2 mGy) imaging scenarios. These characteristics translated to improved low-contrast visualization in anthropomorphic phantoms, demonstrating ≥10% improvement in CNR and extension of the low-dose range for which the detector is input-quantum limited.CONCLUSION:The IGZO-based FPD demonstrated improvements in electronic noise, image lag, and NEQ that translated to measurable improvements in 2D and 3D imaging performance compared to a conventional FPD based on a-Si:H TFTs. The improvements are most beneficial for 2D or 3D imaging scenarios involving low-dose and/or high-frame rate.

Purpose: The accuracy of minimally invasive, intracranial neurosurgery can be challenged by deformation of brain tissue - e.g., up to 10 mm due to egress of cerebrospinal fluid during neuroendoscopic approach. We report an unsupervised, deep learning-based registration framework to resolve such deformations between preoperative MR and intraoperative CT with fast runtime for neurosurgical guidance. Method: The framework incorporates subnetworks for MR and CT image synthesis with a dual-channel registration subnetwork (with synthesis uncertainty providing spatially varying weights on the dual-channel loss) to estimate a diffeomorphic deformation field from both the MR and CT channels. An end-to-end training is proposed that jointly optimizes both the synthesis and registration subnetworks. The proposed framework was investigated using three datasets: (1) paired MR/CT with simulated deformations; (2) paired MR/CT with real deformations; and (3) a neurosurgery dataset with real deformation. Two state-of-the-art methods (Symmetric Normalization and VoxelMorph) were implemented as a basis of comparison, and variations in the proposed dual-channel network were investigated, including single-channel registration, fusion without uncertainty weighting, and conventional sequential training of the synthesis and registration subnetworks. Results: The proposed method achieved: (1) Dice coefficient = 0.82 +/- 0.07 and TRE = 1.2 +/- 0.6 mm on paired MR/CT with simulated deformations; (2) Dice coefficient = 0.83 +/- 0.07 and TRE = 1.4 +/- 0.7 mm on paired MR/CT with real deformations; and (3) Dice = 0.79 +/- 0.13 and TRE = 1.6 +/- 1.0 mm on the neurosurgery dataset with real deformations. The dual-channel registration with uncertainty weighting demonstrated superior performance (e.g., TRE = 1.2 +/- 0.6 mm) compared to single-channel registration (TRE = 1.6 +/- 1.0 mm, p < 0.05 for CT channel and TRE = 1.3 +/- 0.7 mm for MR channel) and dual channel registration without uncertainty weighting (TRE = 1.4 +/- 0.8 mm, p < 0.05). End-to-end training of the synthesis and registration subnetworks also improved performance compared to the conventional sequential training strategy (TRE = 1.3 +/- 0.6 mm). Registration runtime with the proposed network was similar to 3 s. Conclusion: The deformable registration framework based on dual-channel MR/CT registration with spatially varying weights and end-to-end training achieved geometric accuracy and runtime that was superior to state-of-the-art baseline methods and various ablations of the proposed network. The accuracy and runtime of the method may be compatible with the requirements of high-precision neurosurgery. (C) 2021 Elsevier B.V. All rights reserved.

High-precision image-guided neurosurgery – especially in the presence of brain shift – would benefit from intraoperative image quality beyond the conventional contrast-resolution limits of cone-beam CT (CBCT) for visualization of the brain parenchyma, ventricles, and intracranial hemorrhage. Deep neural networks for 3D image reconstruction offer a promising basis for noise and artifact reduction, but generalizability can be challenged in scenarios involving features previously unseen in training data. We propose a 3D deep learning reconstruction framework (termed “DL-Recon”) that integrates learning-based image synthesis with physics-based reconstruction to leverage strengths of each. A 3D conditional GAN was developed to generate synthesized CT from CBCT images. Uncertainty in the synthesis image was estimated in a spatially varying, voxel-wise manner via Monte-Carlo dropout and was shown to correlate with abnormalities or pathology not present in training data. The DL-Recon approach improves the fidelity of the resulting image by combining the synthesized image (“DL-Synthesis”) with physics-based reconstruction (filtered back-projection (FBP) or other approaches) in a manner weighted by uncertainty – i.e., drawing more from the physics-based method in regions where model uncertainty is high. The performance of image synthesis, uncertainty estimation, and DL-Recon was investigated for the first time in real CBCT images of the brain. Variable input to the synthesis network was tested – including uncorrected FBP and precorrection with a simple (constant) scatter estimate – hypothesizing the latter to improve synthesis performance. The resulting uncertainty estimation was evaluated for the first time in real anatomical features not included in training (abnormalities and brain shift). The performance of DL-Recon was evaluated in terms of image uniformity, noise, and soft-tissue contrast-to-noise ratio in comparison to DL-Synthesis and FBP with a comprehensive artifact correction framework. DL-Recon was found to leverage the strengths of the learning-based and physics-based reconstruction approaches, providing a high degree of image uniformity similar to DL-Synthesis while accurately preserving soft-tissue contrast as in artifact-corrected FBP.

Conventional neuro-navigation can be challenged in targeting deep brain structures via transventricular neuroendoscopy due to unresolved geometric error following soft-tissue deformation. Current robot-assisted endoscopy techniques are fairly limited, primarily serving to planned trajectories and provide a stable scope holder. We report the implementation of a robot-assisted ventriculoscopy (RAV) system for 3D reconstruction, registration, and augmentation of the neuroendoscopic scene with intraoperative imaging, enabling guidance even in the presence of tissue deformation and providing visualization of structures beyond the endoscopic field-of-view. Phantom studies were performed to quantitatively evaluate image sampling requirements, registration accuracy, and computational runtime for two reconstruction methods and a variety of clinically relevant ventriculoscope trajectories. A median target registration error of 1.2 mm was achieved with an update rate of 2.34 frames per second, validating the RAV concept and motivating translation to future clinical studies.

BACKGROUND Spinal deformation during surgical intervention (caused by patient positioning and/or correction of malalignment) confounds conventional navigation due to assumptions of rigid transformation. Moreover, the ability to accurately quantify spinal alignment in the operating room would provide assessment of the surgical product via metrics that correlate with clinical outcome. PURPOSE A method for deformable 3D-2D registration of preoperative CT to intraoperative long-length tomosynthesis images is reported for accurate 3D evaluation of device placement in the presence of spinal deformation and automated evaluation of global spinal alignment (GSA). METHODS Long-length tomosynthesis ("Long Film", LF) images were acquired using an O-arm™ imaging system (Medtronic, Minneapolis USA). A deformable 3D-2D patient registration was developed using multi-scale masking (proceeding from the full-length image to local subvolumes about each vertebra) to transform vertebral labels and planning information from preoperative CT to the LF images. Automatic measurement of GSA [Main Thoracic Kyphosis (MThK) and Lumbar Lordosis (LL)] was obtained using a spline fit to registered labels. The "Known-Component Registration" (KC-Reg) method for device registration was adapted to the multi-scale process for 3D device localization from orthogonal LF images. The multi-scale framework was evaluated using a deformable spine phantom in which pedicle screws were inserted, and deformations were induced over a range in LL ∼25-80°. Further validation was carried out in a cadaver study with implanted pedicle screws and a similar range of spinal deformation. The accuracy of patient and device registration was evaluated in terms of 3D translational error and target registration error (TRE), respectively, and the accuracy of automatic GSA measurements were compared to manual annotation. RESULTS Phantom studies demonstrated accurate registration via the multi-scale framework for all vertebral levels in both the neutral and deformed spine: median (interquartile range, IQR) patient registration error was 1.1 mm (0.7-1.9 mm IQR). Automatic measures of MThK and LL agreed with manual delineation within -1.1° ± 2.2° and 0.7° ± 2.0° (mean and standard deviation), respectively. Device registration error was 0.7 mm (0.4-1.0 mm IQR) at the screw tip and 0.9° (1.0°-1.5°) about the screw trajectory. Deformable 3D-2D registration significantly outperformed conventional rigid registration (p < 0.05), which exhibited device registration error of 2.1 mm (0.8-4.1 mm) and 4.1° (1.2°-9.5°). Cadaver studies verified performance under realistic conditions, demonstrating patient registration error of 1.6 mm (0.9-2.1 mm); MThK within -4.2° ± 6.8° and LL within 1.7° ± 3.5°; and device registration error of 0.8 mm (0.5-1.9 mm) and 0.7° (0.4°-1.2°) for the multi-scale deformable method, compared to 2.5 mm (1.0-7.9 mm) and 2.3° (1.6°-8.1°) for rigid registration (p < 0.05). CONCLUSION The deformable 3D-2D registration framework leverages long-length intraoperative imaging to achieve accurate patient and device registration over extended lengths of the spine (up to 64 cm) even with strong anatomical deformation. The method offers a new means for quantitative validation of spinal correction (intraoperative GSA measurement) and 3D verification of device placement in comparison to preoperative images and planning data. This article is protected by copyright. All rights reserved.

Objective. The accuracy of navigation in minimally invasive neurosurgery is often challenged by deep brain deformations (up to 10 mm due to egress of cerebrospinal fluid during neuroendoscopic approach). We propose a deep learning-based deformable registration method to address such deformations between preoperative MR and intraoperative CBCT. Approach. The registration method uses a joint image synthesis and registration network (denoted JSR) to simultaneously synthesize MR and CBCT images to the CT domain and perform CT domain registration using a multi-resolution pyramid. JSR was first trained using a simulated dataset (simulated CBCT and simulated deformations) and then refined on real clinical images via transfer learning. The performance of the multi-resolution JSR was compared to a single-resolution architecture as well as a series of alternative registration methods (symmetric normalization (SyN), VoxelMorph, and image synthesis-based registration methods). Main results. JSR achieved median Dice coefficient (DSC) of 0.69 in deep brain structures and median target registration error (TRE) of 1.94 mm in the simulation dataset, with improvement from single-resolution architecture (median DSC = 0.68 and median TRE = 2.14 mm). Additionally, JSR achieved superior registration compared to alternative methods—e.g. SyN (median DSC = 0.54, median TRE = 2.77 mm), VoxelMorph (median DSC = 0.52, median TRE = 2.66 mm) and provided registration runtime of less than 3 s. Similarly in the clinical dataset, JSR achieved median DSC = 0.72 and median TRE = 2.05 mm. Significance. The multi-resolution JSR network resolved deep brain deformations between MR and CBCT images with performance superior to other state-of-the-art methods. The accuracy and runtime support translation of the method to further clinical studies in high-precision neurosurgery.

Purpose: Recent neurosurgical techniques require accurate targeting of deep-brain structures even in the presence of deformation due to egress of cerebrospinal fluid (CSF) during surgical access. Prior work reported Structure-from-Motion (SfM) based methods for endoscopic guidance using 3D reconstruction. We are developing feature detection and description methods for a real-time 3D endoscopic navigation system using simultaneous localization and mapping (SLAM) to for accurate and near real-time registration. Methods: Feature detectors and descriptors were evaluated in SLAM reconstruction in anthropomorphic phantom studies emulating neuroendoscopy. The experimental system utilized a mobile UR3e robot (Universal Robots, Denmark) and ventriculoscope (Karl Storz, Tuttlingen, Germany) affixed to the end effector as a repeatable ventriculoscopy platform. Experiments were conducted to quantify optimal feature detection parameters in scale-space. Neuroendoscopic images acquired in traversal of the lateral and third ventricles provided a rich feature space of vessels and other structures on ventricular walls supporting feature detection and 3D point-cloud reconstruction. Performance was evaluated in terms of the mean number of features detected per frame and the algorithm runtime. Results: Parameter search in scale-space for feature detection demonstrated the dependence on the mean number of features per image and the points of diminishing return in parameter selection (e.g., the number of octaves and scale levels) and tradeoffs in runtime. Nominal parameters were identified as 3 octaves and 9 scale levels, with a mean number of features detected as 492 and 806 respectively. Conclusions: The system for neuroendoscopic guidance based on SLAM 3D point-cloud reconstruction provided a promising platform for the development of robot-assisted endoscopic neurosurgery. The studies reported in this work provided an essential basis for rigorous selection of parameters for feature detection. Future work aims to further develop the SLAM framework, assess the geometric accuracy of reconstruction, and translate methods to clinical studies.