CryoGAN uses ideas from deep generative adversarial learning to perform image reconstruction in single-particle cryo-electron microscopy (cryo-EM). In this chapter, we begin by introducing single-particle cryo-EM. We then formulate the associated image-reconstruction problem and discuss the main solutions found in the literature. Next, we describe the CryoGAN algorithm and show some representative results. Finally, we discuss what our experiences with Cryo-GAN suggest about the advantages and disadvantages of such deep generative adversarial methods in single-particle cryo-EM and beyond.

The variety of bioimage data and their quality have dramatically increased over the last decade. In parallel, the number of proposed deep learning (DL) models for their analysis grows by the day. Yet, the adequate reuse of published tools by practitioners without DL expertise still raises many practical questions. In this article, we explore four categories of challenges faced by researchers when using supervised DL models in bioimaging applications. We provide examples in which each challenge arises and review the consequences that inadequate decisions may have. We then outline good practices that can be implemented to address the challenges of each category in a scientifically sound way. We provide pointers to the resources that are already available or in active development to help in this endeavor and advocate for the development of further community-driven standards. While primarily intended as a practical tutorial for life scientists, this article also aims at fostering discussions among method developers around the formulation of guidelines for the adequate deployment of DL, with the ultimate goal of accelerating the adoption of novel DL technologies in the biology community.

DeepImageJ is a user-friendly solution that enables the generic use of pre-trained deep learning models for biomedical image analysis in ImageJ. The deepImageJ environment gives access to the largest bioimage repository of pre-trained deep learning models (BioImage Model Zoo). Hence, nonexperts can easily perform common image processing tasks in life-science research with deep learning-based tools including pixel and object classification, instance segmentation, denoising or virtual staining. DeepImageJ is compatible with existing state of the art solutions and it is equipped with utility tools for developers to include new models. Very recently, several training frameworks have adopted the deepImageJ format to deploy their work in one of the most used softwares in the field (ImageJ). Beyond its direct use, we expect deepImageJ to contribute to the broader dissemination and reuse of deep learning models in life sciences applications and bioimage informatics.

A major challenge in single-particle cryo-electron microscopy (cryo-EM) is that the orientations adopted by the 3D particles prior to imaging are unknown; yet, this knowledge is essential for high-resolution reconstruction. We present a method to recover these orientations directly from the acquired set of 2D projections. Our approach consists of two steps: (i) the estimation of distances between pairs of projections, and (ii) the recovery of the orientation of each projection from these distances. In step (i), pairwise distances are estimated by a Siamese neural network trained on synthetic cryo-EM projections from resolved bio-structures. In step (ii), orientations are recovered by minimizing the difference between the distances estimated from the projections and the distances induced by the recovered orientations. We evaluated the method on synthetic cryo-EM datasets. Current results demonstrate that orientations can be accurately recovered from projections that are shifted and corrupted with a high level of noise. The accuracy of the recovery depends on the accuracy of the distance estimator. While not yet deployed in a real experimental setup, the proposed method offers a novel learning-based take on orientation recovery in SPA. Our code is available at https://github.com/JelenaBanjac/protein-reconstruction

We present CryoGAN, a new paradigm for single-particle cryo-EM reconstruction based on unsupervised deep adversarial learning. The major challenge in single-particle cryo-EM is that the imaged particles have unknown poses. Current reconstruction techniques are based on a marginalized maximum-likelihood formulation that requires calculations over the set of all possible poses for each projection image, a computationally demanding procedure. CryoGAN sidesteps this problem by using a generative adversarial network (GAN) to learn the 3D structure that has simulated projections that most closely match the real data in a distributional sense. The architecture of CryoGAN resembles that of standard GAN, with the twist that the generator network is replaced by a model of the cryo-EM image acquisition process. CryoGAN is an unsupervised algorithm that only demands projection images and an estimate of the contrast transfer function parameters. No initial volume estimate or prior training is needed. Moreover, CryoGAN requires minimal user interaction and can provide reconstructions in a matter of hours on a high-end GPU. In addition, we provide sound mathematical guarantees on the recovery of the correct structure. CryoGAN currently achieves a 8.6 Å resolution on a realistic synthetic dataset. Preliminary results on realβ-galactosidase data demonstrate CryoGAN’s ability to exploit data statistics under standard experimental imaging conditions. We believe that this paradigm opens the door to a family of novel likelihood-free algorithms for cryo-EM reconstruction.

The current shift towards computational imaging has made reconstruction procedures an integral part of many advanced imaging systems. A consequence is that imaging scientists now commonly require efficient and reliable computational tools for solving their inverse problems. To this end, we recently developed GlobalBioIm, an open-source Matlab library that standardizes the resolution of a wide range of imaging problems [1]. This toolbox gives access to cutting-edge reconstruction algorithms, and can be extended to new modalities and methods by combining elementary modules. The versatility and efficiency of GlobalBioIm have been highlighted in a series of recent high-impact works [2-4]. Driven by these encouraging applications, we have devoted our efforts towards improving the usability of GlobalBioIm by those with limited expertise in inverse problems and optimization theory. The outcome is a new user-friendly Matlab interface (Figure 1) that allows non-experts to intuitively build tailored reconstruction algorithms with minimal effort. Figure 1: The new GlobalBioIm GUI (left) can be used to solve various inverse problems (right).

Single-particle cryo-electron microscopy (cryo-EM) reconstructs the three-dimensional (3D) structure of bio-molecules from a large set of 2D projection images with random and unknown orientations. A crucial step in the single-particle cryo-EM pipeline is 3D refinement, which resolves a high-resolution 3D structure from an initial approximate volume by refining the estimation of the orientation of each projection. In this work, we propose a new approach that refines the projection angles on the continuum. We formulate the optimization problem over the density map and the orientations jointly. The density map is updated using the efficient alternating-direction method of multipliers, while the orientations are updated through a semi-coordinate-wise gradient descent for which we provide an explicit derivation of the gradient. Our method eliminates the requirement for a fine discretization of the orientation space and does away with the classical but computationally expensive template-matching step. Numerical results demonstrate the feasibility and performance of our approach compared to several baselines.

Imaging scientists nowadays commonly relies on the deployment of sophisticated algorithms to recover an object of interest x ∈ R from measurements y ∈ R . These quantities are linked according to

Thanks to recent advances in signal processing, the interest for fast ℓ1-regularized reconstruction algorithms in cryo-electron microscopy (cryo-EM) has intensified. The approaches based on the alternating-direction of multipliers method (ADMM) are particularly well-suited due to the prime convergence speed and flexibility of use of this algorithm. Yet, the standard ADMM scheme still relies on a nested conjugate gradient (CG) to solve the linear step in its alternating-minimization procedure, which can be costly when handling large-scale problems. In this work, we present an inner-loop-free ADMM algorithm for 3D reconstruction in cryo-EM. By using an appropriate splitting scheme, we are able to avoid the use of CG for solving the linear step. This leads to a substantial increase in algorithmic speed, as demonstrated by our experiments.

GlobalBioIm is an open-source MATLAB ® library for solving inverse problems. The library capitalizes on the strong commonalities between forward models to standardize the resolution of a wide range of imaging inverse problems. Endowed with an operator-algebra mechanism, GlobalBioIm allows one to easily solve inverse problems by combining elementary modules in a lego-like fashion. This user-friendly toolbox gives access to cutting-edge reconstruction algorithms, while its high modularity makes it easily extensible to new modalities and novel reconstruction methods. We expect GlobalBioIm to respond to the needs of imaging scientists looking for reliable and easy-to-use computational tools for solving their inverse problems. In this paper, we present in detail the structure and main features of the library. We also illustrate its flexibility with examples from multichannel deconvolution microscopy.

Cryo-electron microscopy (Cryo-EM) is a popular imaging modality used to visualize a wide range of bio-molecules in their 3D form. The goal in Cryo-EM is to reconstruct the 3D density map of a molecule from projection images taken from random and unknown orientations. A critical step in the Cryo-EM pipeline is 3D refinement. In this procedure, an initial 3D map and a set of estimated projection orientations is refined to obtain higher resolution maps. State-of-the-art refinement techniques rely on projection matching steps in order to refine the initial projection orientations. Unfortunately projection matching is computationally inefficient and it requires a finite discretization of the space of orientations. To avoid repeated projection matching steps, in this work we consider the orientation variables in their continuous form. This enables us to formulate the refinement problem as a joint optimization problem that refines the underlying density map and orientations. We use alternating direction method of multipliers (ADMM) and gradient descent steps in order to update the density map and the orientations, respectively. Our results and their comparison with several baselines demonstrate the feasibility and performance of the proposed refinement framework.

We present a multiscale reconstruction framework for single-particle analysis (SPA). The representation of three-dimensional (3D) objects with scaled basis functions permits the reconstruction of volumes at any desired scale in the real-space. This multiscale approach generates interesting opportunities in SPA for the stabilization of the initial volume problem or the 3D iterative refinement procedure. In particular, we show that reconstructions performed at coarse scale are more robust to angular errors and permit gains in computational speed. A key component of the proposed iterative scheme is its fast implementation. The costly step of reconstruction, which was previously hindering the use of advanced iterative methods in SPA, is formulated as a discrete convolution with a cost that does not depend on the number of projection directions. The inclusion of the contrast transfer function inside the imaging matrix is also done at no extra computational cost. By permitting full 3D regularization, the framework is by itself a robust alternative to direct methods for performing reconstruction in adverse imaging conditions (e.g., heavy noise, large angular misassignments, low number of projections). We present reconstructions obtained at different scales from a dataset of the 2015/2016 EMDataBank Map Challenge. The algorithm has been implemented in the Scipion package.

We designed a complete acquisition-reconstruction framework to reduce the radiation dosage in 3D scanning transmission electron microscopy (STEM). Projection measurements are acquired by randomly scanning a subset of pixels at every tilt-view (i.e., random-beam STEM or “RB-STEM”). High-quality images are then recovered from the randomly downsampled measurements through a regularized tomographic reconstruction framework. By fulfilling the compressed sensing requirements, the proposed approach improves the reconstruction of heavily-downsampled RB-STEM measurements over the current state-of-the-art technique. This development opens new perspectives in the search for methods permitting lower-dose 3D STEM imaging of electron-sensitive samples without degrading the quality of the reconstructed volume. A Matlab code implementing the proposed reconstruction algorithm has been made available online.

Files and metadata associated with the EMDataBank/Unified Data Resource for 3DEM 2015/2016 Map Challenge hosted at challenges.emdatabank.org are deposited. All members of the Scientific Community--at all levels of experience--were invited to participate as Challengers, and/or as Assessors. Seven benchmark raw image datasets were selected for the challenge. Six are selected from recently described single particle structure determinations with image data collected as multi-frame movies; one is based on simulated (in silico) images. All of the raw image datasets are archived at pdbe.org/empiar. 27 Challengers created 66 single particle reconstructions from the targets, and then uploaded their results with associated details. 15 of the reconstructions were calculated using the SDSC Gordon supercomputer. This map challenge was one of two community-wide challenges sponsored by EMDataBank in 2015/2016 to critically evaluate 3DEM methods that are coming into use, with the ultimate goal of developing validation criteria associated with every 3DEM map and map-derived model.

Images in fluorescence microscopy are inherently blurred due to the limit of diffraction of light. The purpose of deconvolution microscopy is to compensate numerically for this degradation. Deconvolution is widely used to restore fine details of 3D biological samples. Unfortunately, dealing with deconvolution tools is not straightforward. Among others, end users have to select the appropriate algorithm, calibration and parametrization, while potentially facing demanding computational tasks. To make deconvolution more accessible, we have developed a practical platform for deconvolution microscopy called DeconvolutionLab. Freely distributed, DeconvolutionLab hosts standard algorithms for 3D microscopy deconvolution and drives them through a user-oriented interface. In this paper, we take advantage of the release of DeconvolutionLab2 to provide a complete description of the software package and its built-in deconvolution algorithms. We examine several standard algorithms used in deconvolution microscopy, notably: Regularized inverse filter, Tikhonov regularization, Landweber, Tikhonov-Miller, Richardson-Lucy, and fast iterative shrinkage-thresholding. We evaluate these methods over large 3D microscopy images using simulated datasets and real experimental images. We distinguish the algorithms in terms of image quality, performance, usability and computational requirements. Our presentation is completed with a discussion of recent trends in deconvolution, inspired by the results of the Grand Challenge on deconvolution microscopy that was recently organized.

We present a unifying framework for the development of state-of-the-art reconstruction algorithms in computational optics with a clear separation between the physical (forward model) and signal-related (regularization, incorporation of prior constraints) aspects of the problem. The pillars of our formulation are: (i) an operator algebra with its set of fast linear solvers, (ii) a variational derivation of reconstruction methods, and (iii) a suite of efficient numerical tools for the resolution of large-scale optimization problems. These core technologies are incorporated into a modular software library featuring the key components for the implementation and testing of iterative reconstruction algorithms. The concept is illustrated with concrete examples in 3D deconvolution microscopy and lenseless imaging.

Electron tomography is a fantastic tool for deciphering the structural information of complex 3D samples. During the last years, several tools have been developped to improve the 3D reconstruction quality of thick specimens. The direct detector cameras have incredibly increased the SNR and resolution of thin samples 2D projections, bringing electron microscopy resolution at the level of the one of X‐ray diffraction studies. However, the study of thick biological samples in tomography still suffers from the too important electron dose one has to use in order to retrieve high quality images and reconstructions. It has been shown that STEM tomography can generate more accurate reconstructions than TEM tomography while preserving better the sample integrity. Previous uses of compressive sensing enabled the reduction of tilt‐angles in tomography studies, unveiling electron dose reduction. Here, we push further the electron dose reduction thanks to a more effective compressive sensing method which uses incomplete images as incoherent data. The generation of incomplete images being performed at the microscope during the acquisition process where the beam randomly scans the surface of the sample.