We present a geometry-aware method for heterogeneous single-particle cryogenic electron microscopy (cryo-EM) reconstruction that predicts atomic backbone conformations. To incorporate protein-structure priors, we represent the backbone as a graph and use a graph neural network (GNN) autodecoder that maps per-image latent variables to 3D displacements of a template conformation. The objective combines a data-discrepancy term based on a differentiable cryo-EM forward model with geometric regularization, and it supports unknown orientations via ellipsoidal support lifting (ESL) pose estimation. On synthetic datasets derived from molecular dynamics trajectories, the proposed GNN achieves higher accuracy compared to a multilayer perceptron (MLP) of comparable size, highlighting the benefits of a geometry-informed inductive bias.

We introduce a method for constructing predictive models of non cyclic physical processes directly from time-series data, without assuming an underlying differential equation. The observations define a discrete evolution rule whose recurrent behaviour captures the essential dynamics of the process. Analysing this behaviour across multiple geometric scales leads to probabilistic models in the form of Markov chains. Hyperbolicity criteria identify when these models provide a consistent statistical description of the data. The method is inspired by, and illustrated through, the analysis of a biological imaging data set referred to as the Cell Process.

We present a novel multiscale algorithm for directly recovering the atomic model structure of a protein from single-particle cryo-EM data. Our algorithm is able to estimate protein structures to state-of-the-art accuracy for high-noise and low-contrast data. It is also robust to misspecifications in the TEM image formation model. These desirable properties are primarily due to the use of an explicit representation of the protein backbone in terms of bonds, torsion angles and bond angles, which supplies rich prior information to the structure recovery process. We apply our method on three protein cryo-EM datasets, generated using an electron microscope digital twin, and show that using a multiscale approach yields an improvement of the root-mean-square deviation (RMSD) and template modelling (TM) scores with respect to the ground truth. Furthermore, there is evidence that larger-scale structures are being prioritised with the multiscale algorithm, which reduces the possibility of convergence to bad local minima.

Single-particle cryo-electron microscopy images a macromolecule as many noisy tomographic projections of its electrostatic potential. We reconstruct the protein backbone directly from such projections, as an atomic point cloud, without the intermediate step of reconstructing the 3D electrostatic potential map. We formulate this as an indirect shape matching problem: a point-cloud template of the backbone is deformed until its simulated projections agree with the data, with the structure observed only through the imaging operator. The deformation is computed via a gradient flow on a Lie group, and we derive the resulting framework in a general geometric setting before adapting it for single-particle cryo-electron microscopy. On synthetic data, we recover single- and multichain proteins and capture conformational transitions.

Recovering physical properties of objects in motion is a core task across scientific and industrial applications. When the relative motion between the object and the sensing apparatus provides sufficient angular coverage, Computerized Tomography offers a powerful means of reconstruction. For such scenarios, we propose a parametric spatiotemporal model applied to Gaussian Mixture Models (GMM), in which each constituent Gaussian is parameterized by its own angular velocity, projectile motion, and geometry. GMM are a suitable means of reconstruction because they (i) admit accurate approximations in object space and (ii) have a closed form expression under the ray transform; enabling efficient forward predictions and exact gradient computations in data space. By decoupling the reconstruction problem into two sub-inverse problems, we characterize solutions as minimizers of task-specific objective functions that are derived and solved by utilizing the properties of (ii). The resulting algorithm we provide is applicable to objects in Euclidean space of arbitrary dimension. We validate the method on a simulated 2D problem, achieving accurate reconstruction of a 5-Gaussian GMM with intersecting trajectories. This also provides a foundation for further experimentation in settings with noisy data, 3D objects, and non-rigid body dynamics.

Data-driven discovery of dynamics in biological systems allows for better observation and characterization of processes, such as calcium signaling in cell culture. Recent advancements in techniques allow the exploration of previously unattainable insights of dynamical systems, such as the Sparse Identification of Non-Linear Dynamics (SINDy), overcoming the limitations of more classic methodologies. The latter requires some prior knowledge of an effective library of candidate terms, which is not realistic for a real case study. Using inspiration from fields like traffic density estimation and control theory, we propose a methodology for characterization and performance analysis of calcium delivery in a family of cells. In this work, we compare the performance of the Constrained Regularized Least-Squares Method (CRLSM) and Physics-Informed Neural Networks (PINN) for system identification and parameter discovery for governing ordinary differential equations (ODEs). The CRLSM achieves a fairly good parameter estimate and a good data fit when using the learned parameters in the Consensus problem. On the other hand, despite the initial hypothesis, PINNs fail to match the CRLSM performance and, under the current configuration, do not provide fair parameter estimation. However, we have only studied a limited number of PINN architectures, and it is expected that additional hyperparameter tuning, as well as uncertainty quantification, could significantly improve the performance in future works.

Learned image reconstruction has become a pillar in computational imaging and inverse problems. Among the most successful approaches are learned iterative networks, which are formulated by unrolling classical iterative optimisation algorithms for solving variational problems. While the underlying algorithm is usually formulated in the functional analytic setting, learned approaches are often viewed as purely discrete. In this chapter we present a unified operator view for learned iterative networks. Specifically, we formulate a learned reconstruction operator, defining how to compute, and separately the learning problem, which defines what to compute. In this setting we present common approaches and show that many approaches are closely related in their core. We review linear as well as nonlinear inverse problems in this framework and present a short numerical study to conclude.

Objective. Proton therapy treatment planning currently needs to account for relatively large range uncertainty margins primarily due to the semi-empirical calibration of the treatment planning x-ray computed tomography (CT) to proton stopping power relative to water (RSP). Proton radiography enables direct measurement of integral RSP, offering potential to improve calibration and reduce these uncertainties. Approach. Three deep neural network architectures are trained to infer a patient RSP map using the treatment planning CT and proton radiographies, assuming straight proton trajectories. The best suited architecture is then evaluated on more realistic, clinical-like data generated with Monte Carlo simulations. Three idealized proton imaging detectors are simulated: single particle tracking (SPT), a pixelated energy-resolving imager (PERI) and a proton-integrating detector (PID). Main results. The learned primal dual (LPD) architetcure performs best in the simplified imaging scenario. In the more realistic scenario, the initial calibration error (median absolute percentage error of 2.24% across the test set) is reduced using only two projections across all detector types. SPT and PERI reach similar performance (1.10%/1.12%), followed by PID (1.30%). Restricting the LPD to the calibration task by incorporating prior knowledge of the functional relationship of Hounsfield Units (HUs) and RSP further improves calibration performance. For SPT, conventional optimization on detector data acquired for individual protons outperformed the data-driven method (0.16% vs. 1.10%). However, for PERI and PID, the data-driven approach (1.12%/1.30%) slightly outperformed conventional optimization (1.63%/1.72%). Significance. To our knowledge, this is the first study to apply a deep learning-based approach fusing proton radiographies and treatment planning CT data for improved RSP calibration. The method achieves lower calibration errors than idealized, conventional calibration curve optimization on PERIs and PIDs-detector types that offer a promising path for clinical adoption due to their lower complexity and cost compared to SPT systems.

We address recovery of the three-dimensional backbone structure of single polypeptide proteins from single-particle cryo-electron microscopy (Cryo-SPA) data. Cryo-SPA produces noisy tomographic projections of electrostatic potentials of macromolecules. From these projections, we use methods from shape analysis to recover the three-dimensional backbone structure. Thus, we view the reconstruction problem as an indirect matching problem, where a point cloud representation of the protein backbone is deformed to match two-dimensional tomography data. The deformations are obtained via the action of a matrix Lie group. By selecting a deformation energy, the optimality conditions are obtained, which lead to computational algorithms for optimal deformations. We showcase our approach on synthetic data, for which we recover the three-dimensional structure of the backbone.

In the wood industry, logs are commonly quality screened by discrete X-ray scans on a moving conveyor belt from a few source positions. Typically, the measurements are obtained in a single two-dimensional (2D) plane (a "slice") by a sequential scanning geometry. The data from each slice alone does not carry sufficient information for a three-dimensional tomographic reconstruction in which biological features of interest in the log are well preserved. In the present work, we propose a learned iterative reconstruction method based on the Learned Primal-Dual neural network, suited for sequential scanning geometries. Our method accumulates information between neighbouring slices, instead of only accounting for single slices during reconstruction. Evaluations were performed by training U-Nets on segmentation of knots (branches), which are crucial features in wood processing. Our quantitative and qualitative evaluations show that with as few as five source positions our method yields reconstructions of logs that are sufficiently accurate to identify biological features like knots (branches), heartwood and sapwood.

We propose a learned precomputation for the heterogeneous multiscale method (HMM) for rough-wall Stokes flow. A Fourier neural operator is used to approximate local averages over microscopic subsets of the flow, which allows to compute an effective slip length of the fluid away from the roughness. The network is designed to map from the local wall geometry to the Riesz representors for the corresponding local flow averages. With such a parameterisation, the network only depends on the local wall geometry and as such can be trained independent of boundary conditions. We perform a detailed theoretical analysis of the statistical error propagation, and prove that under suitable regularity and scaling assumptions, a bounded training loss leads to a bounded error in the resulting macroscopic flow. We then demonstrate on a family of test problems that the learned precomputation performs stably with respect to the scale of the roughness. The accuracy in the HMM solution for the macroscopic flow is comparable to when the local (micro) problems are solved using a classical approach, while the computational cost of solving the micro problems is significantly reduced.

Ensuring proper generalization is a critical challenge in applying data-driven methods for solving inverse problems in imaging, as neural networks reconstructing an image must perform well across varied datasets and acquisition geometries. In X-ray Computed Tomography (CT), convolutional neural networks (CNNs) are widely used to filter the projection data but are ill-suited for this task as they apply grid-based convolutions to the sinogram, which inherently lies on a line manifold, not a regular grid. The CNNs, unaware of the geometry, are implicitly tied to it and require an excessive amount of parameters as they must infer the relations between measurements from the data rather than from prior information. The contribution of this paper is twofold. First, we introduce a graph data structure to represent CT acquisition geometries and tomographic data, providing a detailed explanation of the graph's structure for circular, cone-beam geometries. Second, we propose GLM, a hybrid neural network architecture that leverages both graph and grid convolutions to process tomographic data. We demonstrate that GLM outperforms CNNs when performance is quantified in terms of structural similarity and peak signal-to-noise ratio, despite the fact that GLM uses only a fraction of the trainable parameters. Compared to CNNs, GLM also requires significantly less training time and memory, and its memory requirements scale better. Crucially, GLM demonstrates robust generalization to unseen variations in the acquisition geometry, like when training only on fully sampled CT data and then testing on sparse-view CT data.

Direct reconstruction of macromolecular structures from cryogenic electron microscopy (Cryo-EM) data has shown to be a challenge both in the homogeneous and heterogeneous setting. In this work we propose a new direct reconstruction method based on a combination of recent developments on protein geometry and orientation estimation. Even though this method is set up for the homogeneous setting, we aim to gain insight into challenges that atomistic methods have been facing for the heterogeneous case. In numerical experiments we observe that the method is able to recover the structure to almost inter-atomic resolution from as few as 100 2D Cryo-EM images due to the strong bias the regularizer gives. We conclude this work with a discussion on how the obtained results indicate possibilities and challenges for the generalization to the heterogeneous case.

AbstractBackgroundEvaluating AI-based segmentation models primarily relies on quantitative metrics, but it remains unclear if this approach leads to practical, clinically applicable tools.PurposeTo create a systematic framework for evaluating the performance of segmentation models using clinically relevant criteria.Materials and MethodsWe developed the AUGMENT framework (Assessing Utility of seGMENtation Tools), based on a structured classification of main categories of error in segmentation tasks. To evaluate the framework, we assembled a team of 20 clinicians covering a broad range of radiological expertise and analysed the challenging task of segmenting metastatic ovarian cancer using AI. We used three evaluation methods: (i) Dice Similarity Coefficient (DSC), (ii) visual Turing test, assessing 429 segmented disease-sites on 80 CT scans from the Cancer Imaging Atlas), and (iii) AUGMENT framework, where 3 radiologists and the AI-model created segmentations of 784 separate disease sites on 27 CT scans from a multi-institution dataset.ResultsThe AI model had modest technical performance (DSC=72±19 for the pelvic and ovarian disease, and 64±24 for omental disease), and it failed the visual Turing test. However, the AUGMENT framework revealed that (i) the AI model produced segmentations of the same quality as radiologists (p=.46), and (ii) it enabled radiologists to produce human+AI collaborative segmentations of significantly higher quality (p=<.001) and in significantly less time (p=<.001).ConclusionQuantitative performance metrics of segmentation algorithms can mask their clinical utility. The AUGMENT framework enables the systematic identification of clinically usable AI-models and highlights the importance of assessing the interaction between AI tools and radiologists.Summary statementOur framework, called AUGMENT, provides an objective assessment of the clinical utility of segmentation algorithms based on well-established error categories.Key resultsCombining quantitative metrics with qualitative information on performance from domain experts whose work is impacted by an algorithm’s use is a more accurate, transparent and trustworthy way of appraising an algorithm than using quantitative metrics alone.The AUGMENT framework captures clinical utility in terms of segmentation quality and human+AI complementarity even in algorithms with modest technical segmentation performance.AUGMENT might have utility during the development and validation process, including in segmentation challenges, for those seeking clinical translation, and to audit model performance after integration into clinical practice.

Reconstructing images using Computed Tomography (CT) in an industrial context leads to specific challenges that differ from those encountered in other areas, such as clinical CT. Indeed, non-destructive testing with industrial CT will often involve scanning multiple similar objects while maintaining high throughput, requiring short scanning times, which is not a relevant concern in clinical CT. Under-sampling the tomographic data (sinograms) is a natural way to reduce the scanning time at the cost of image quality since the latter depends on the number of measurements. In such a scenario, post-processing techniques are required to compensate for the image artifacts induced by the sinogram sparsity. We introduce the Self-supervised Denoiser Framework (SDF), a self-supervised training method that leverages pre-training on highly sampled sinogram data to enhance the quality of images reconstructed from undersampled sinogram data. The main contribution of SDF is that it proposes to train an image denoiser in the sinogram space by setting the learning task as the prediction of one sinogram subset from another. As such, it does not require ground-truth image data, leverages the abundant data modality in CT, the sinogram, and can drastically enhance the quality of images reconstructed from a fraction of the measurements. We demonstrate that SDF produces better image quality, in terms of peak signal-to-noise ratio, than other analytical and self-supervised frameworks in both 2D fan-beam or 3D cone-beam CT settings. Moreover, we show that the enhancement provided by SDF carries over when fine-tuning the image denoiser on a few examples, making it a suitable pre-training technique in a context where there is little high-quality image data. Our results are established on experimental datasets, making SDF a strong candidate for being the building block of foundational image-enhancement models in CT.

Deep learning based computed tomography (CT) reconstruction has demonstrated outstanding performance on simulated 2D low-dose CT data. This applies in particular to domain adapted neural networks, which incorporate a handcrafted physics model for CT imaging. Empirical evidence shows that employing such architectures reduces the demand for training data and improves upon generalisation. However, their training requires large computational resources that quickly become prohibitive in 3D helical CT, which is the most common acquisition geometry used for medical imaging. Furthermore, clinical data also comes with other challenges not accounted for in simulations, like errors in flux measurement, resolution mismatch and, most importantly, the absence of the real ground truth. The necessity to have a computationally feasible training combined with the need to address these issues has made it difficult to evaluate deep learning based reconstruction on clinical 3D helical CT. This paper modifies a domain adapted neural network architecture, the Learned Primal-Dual (LPD), so that it can be trained and applied to reconstruction in this setting. We achieve this by splitting the helical trajectory into sections and applying the unrolled LPD iterations to those sections sequentially. To the best of our knowledge, this work is the first to apply an unrolled deep learning architecture for reconstruction on full-sized clinical data, like those in the Low dose CT image and projection data set (LDCT). Moreover, training and testing is done on a single GPU card with 24GB of memory.

An increasingly common viewpoint is that protein dynamics data sets reside in a non-linear subspace of low conformational energy. Ideal data analysis tools for such data sets should therefore account for such non-linear geometry. The Riemannian geometry setting can be suitable for a variety of reasons. First, it comes with a rich structure to account for a wide range of geometries that can be modelled after an energy landscape. Second, many standard data analysis tools initially developed for data in Euclidean space can also be generalised to data on a Riemannian manifold. In the context of protein dynamics, a conceptual challenge comes from the lack of a suitable smooth manifold and the lack of guidelines for constructing a smooth Riemannian structure based on an energy landscape. In addition, computational feasibility in computing geodesics and related mappings poses a major challenge. This work considers these challenges. The first part of the paper develops a novel local approximation technique for computing geodesics and related mappings on Riemannian manifolds in a computationally feasible manner. The second part constructs a smooth manifold of point clouds modulo rigid body group actions and a Riemannian structure that is based on an energy landscape for protein conformations. The resulting Riemannian geometry is tested on several data analysis tasks relevant for protein dynamics data. It performs exceptionally well on coarse-grained molecular dynamics simulated data. In particular, the geodesics with given start- and end-points approximately recover corresponding molecular dynamics trajectories for proteins that undergo relatively ordered transitions with medium sized deformations. The Riemannian protein geometry also gives physically realistic summary statistics and retrieves the underlying dimension even for large-sized deformations within seconds on a laptop.