Advances in robotic automation, high-performance computing (HPC), and artificial intelligence (AI) encourage us to conceive of science factories: large, general-purpose computation- and AI-enabled self-driving laboratories (SDLs) with the generality and scale needed both to tackle large discovery problems and to support thousands of scientists. Science factories require modular hardware and software that can be replicated for scale and (re)configured to support many applications. To this end, we propose a prototype modular science factory architecture in which reconfigurable modules encapsulating scientific instruments are linked with manipulators to form workcells, that can themselves be combined to form larger assemblages, and linked with distributed computing for simulation, AI model training and inference, and related tasks. Workflows that perform sets of actions on modules can be specified, and various applications, comprising workflows plus associated computational and data manipulation steps, can be run concurrently. We report on our experiences prototyping this architecture and applying it in experiments involving 15 different robotic apparatus, five applications (one in education, two in biology, two in materials), and a variety of workflows, across four laboratories. We describe the reuse of modules, workcells, and workflows in different applications, the migration of applications between workcells, and the use of digital twins, and suggest directions for future work aimed at yet more generality and scalability. Code and data are available at https://ad-sdl.github.io/wei2023 and in the Supplementary Information

Self-driving labs (SDLs) leverage combinations of artificial intelligence, automation, and advanced computing to accelerate scientific discovery.

Digital technology presents us with new and compelling opportunities for discovery when focused on the world's natural history collections. The outstanding barrier to applying existing and forthcoming computational methods for large-scale study of this important resource is that it is (largely) not yet in the digital realm. Without development of new and much faster methods for digitizing objects in these collections, it will be a long time before these data are available in digital form. For example, methods that are currently employed for capturing, cataloguing, and indexing pinned insect specimen data will require many tens of years or more to process collections with millions of dry specimens, and so we need to develop a much faster pipeline. In this paper we describe a capture system capable of collecting and archiving the imagery necessary to digitize a collection of circa 4.5 million specimens in one or two years of production operation. To minimize the time required to digitize each specimen, we have proposed (Hereld et al. 2017) developing multi-camera systems to capture the pinned insect and its accompanying labels from many angles in a single exposure. Using a sampling (21 randomly drawn drawers, totalling 5178 insects) of the 4.5 million specimens in the collection at the Field Museum of Natural History, we estimated that a large fraction of that collection (97.6% +/- 2.2%) consists of pinned insects with labels that are visible from one angle or another without requiring adjustment or removal of elements on the pin. In this situation a multi-camera system with enough angular coverage could provide imagery for reconstructing virtual labels from fragmentary views taken from different directions. Agarwal et al. (2018) demonstrated a method for combining these multiple views into a virtual label that could be transcribed by automated optical character recognition software. We have now designed, built and tested a prototype snapshot 3D digitization station to allow rapid capture of multi-view imagery for automated capture of pinned insect specimens and labels. It consists of twelve very small and light 8-megapixel cameras (Fig. 1), each controlled by a small dedicated computer. The cameras are arrayed around the target volume, six on each side of the sample feed path. Their positions and orientations are fixed by a 3D-printed scaffolding designed for the purpose. The twelve camera controllers and a master computer are connected to a dedicated high-speed data network over which all of the coordinating control signals and returning images and metadata are passed. The system is integrated with a high-performance object store that includes a database for metadata and the archived images comprising each snapshot. The system is designed so that it can be readily extended to include additional or different sensors. The station is meant to be fed with specimens by a conveyor belt whose motion is coordinated with the exposure of the multi-view snapshots. In order to test the performance of the system we added a recirculating specimen feeder designed expressly for this experiment. With it integrated into the system in place of a conventional conveyor belt we are able to provide a continuous stream of targets for the digitization system to facilitate long tests of its performance and robustness. We demonstrated the ability to capture data at a peak rate of 1400 specimens per hour and an average rate of 1000 specimens per hour over the course of a sustained 6 hour run. The dataset (Hereld and Ferrier 2018) collected in this experiment provides fodder for the further development of algorithms for the offline reconstruction and automatic transcription of the label contents.

Accurate, precise, and rapid particle tracking in three dimensions remains a challenge; yet, its achievement will significantly enhance our understanding of living systems. We developed a multifocal microscopy (MFM) that allows snapshot acquisition of the imaging data, and an associated image processing approach, that together allow simultaneous 3D tracking of many fluorescent particles with nanoscale resolution. The 3D tracking was validated by measuring a known trajectory of a fluorescent bead with an axial accuracy of 19 nm through an image depth (axial range) of 3 pm and 4 nm precision of axial localization through an image depth of 4 pm. A second test obtained a uniform axial probability distribution and Brownian dynamics of beads diffusing in solution. We also validated the MFM approach by imaging fluorescent beads immobilized in gels and comparing the 3D localizations to their "ground truth" positions obtained from a confocal microscopy z-stack of finely spaced images. Finally, we applied our MFM and image processing approach to obtain 3D trajectories of insulin granules in pseudoislets of MIN6 cells to demonstrate its compatibility with complex biological systems. Our study demonstrates that multifocal microscopy allows rapid (video rate) and simultaneous 3D tracking of many "particles" with nanoscale accuracy and precision in a wide range of systems, including over spatial scales relevant to whole live cells.

Volumetric biological imaging often involves compromising high temporal resolution at the expense of high spatial resolution when popular scanning methods are used to capture 3D information. We introduce an integrated experimental and image reconstruction method for capturing dynamic 3D fluorescent extended objects as a series of synchronously measured 3D snapshots taken at the frame rate of the imaging camera. We employ multifocal microscopy (MFM) to simultaneously image at 25 focal planes and process this depth-encoded image to recover the 3D structure of extended objects, such as bacteria, using a sparsity-based reconstruction approach. The combined experimental and computational method produces image quality similar to confocal microscopy in a fraction of the acquisition time. In addition, our computational image reconstruction approach allows a simplified MFM optical design by correcting aberrations using the measured response to point sources. This "compressive" MFM acquisition and reconstruction method, where an image volume with roughly 8 million voxels is recovered from a single 1-megapixel captured image, enables straightforward study of dynamic processes in 3D, and as a simultaneous snapshot advances the state of the art in dynamic 3D microscopy.

Rapid and accurate volumetric imaging remains a challenge, yet has the potential to enhance understanding of cell function. We developed and used a multifocal microscope (MFM) for 3D snapshot imaging to allow 3D tracking of insulin granules labeled with mCherry in MIN6 cells. MFM employs a special diffractive optical element (DOE) to simultaneously image multiple focal planes. This simultaneous acquisition of information determines the 3D location of single objects at a speed only limited by the frame rate of array detector . We validated the accuracy of MFM imaging and tracking with fluorescence beads; the 3D positions and trajectories of single fluorescence beads can be determined accurately over a wide range of spatial and temporal scales. The 3D positions and trajectories of single insulin granules in a 3.2 micro meter deep volume were determined with imaging processing that combines 3D decovolution, shift correction, and finally tracking using the Imaris software package. We find that the motion of the granules is super-diffusive, but less so in 3D than 2D for cells grown on coverslip surfaces, suggesting an anisotropy in the cytoskeleton (e.g. microtubules and action).

Accurate and rapid particle tracking is essential for addressing many research problems in single molecule and cellular biophysics and colloidal soft condensed matter physics. We developed a novel three-dimensional interferometric fluorescent particle tracking approach that does not require any sample scanning. By periodically shifting the interferometer phase, the information stored in the interference pattern of the emitted light allows localizing particles positions with nanometer resolution. This tracking protocol was demonstrated by measuring a known trajectory of a fluorescent bead with sub-5 nm axial localization error at 5 Hz. The interferometric microscopy was used to track the RecA protein in Bacillus subtilis bacteria to demonstrate its compatibility with biological systems.

We present a Bayesian approach for 3D image reconstruction of an extended object imaged with multi-focus microscopy (MFM). MFM simultaneously captures multiple sub-images of different focal planes to provide 3D information of the sample. The naive method to reconstruct the object is to stack the sub-images along the z-axis, but the result suffers from poor resolution in the z-axis. The maximum a posteriori framework provides a way to reconstruct a 3D image according to its observation model and prior knowledge. It jointly estimates the 3D image and the model parameters. Experimental results with synthetic and real experimental data show that it enables the high-quality 3D reconstruction of an extended object from MFM.

We present a primal-dual interior point method (IPM) with a novel preconditioner to solve the ℓ 1 -norm regularized least square problem for nonnegative sparse signal reconstruction. IPM is a second-order method that uses both gradient and Hessian information to compute effective search directions and achieve super-linear convergence rates. It therefore requires many fewer iterations than first-order methods such as iterative shrinkage/thresholding algorithms (ISTA) that only achieve sub-linear convergence rates. However, each iteration of IPM is more expensive than in ISTA because it needs to evaluate an inverse of a Hessian matrix to compute the Newton direction. We propose to approximate each Hessian matrix by a diagonal matrix plus a rank-one matrix. This approximation matrix is easily invertible using the Sherman-Morrison formula, and is used as a novel preconditioner in a preconditioned conjugate gradient method to compute a truncated Newton direction. We demonstrate the efficiency of our algorithm in compressive 3D volumetric image reconstruction. Numerical experiments show favorable results of our method in comparison with previous interior point based and iterative shrinkage/thresholding based algorithms.

An interferometric fluorescent microscope and a novel theoretic image reconstruction approach were developed and used to obtain super-resolution images of live biological samples and to enable dynamic real time tracking. The tracking utilizes the information stored in the interference pattern of both the illuminating incoherent light and the emitted light. By periodically shifting the interferometer phase and a phase retrieval algorithm we obtain information that allow localization with sub-2 nm axial resolution at 5 Hz.

Despite recent advances, high performance single-shot 3D microscopy remains an elusive task. By introducing designed diffractive optical elements (DOEs), one is capable of converting a microscope into a 3D "kaleidoscope," in which case the snapshot image consists of an array of tiles and each tile focuses on different depths. However, the acquired multifocal microscopic (MFM) image suffers from multiple sources of degradation, which prevents MFM from further applications. We propose a unifying computational framework which simplifies the imaging system and achieves 3D reconstruction via computation. Our optical configuration omits optical elements for correcting chromatic aberrations and redesigns the multifocal grating to enlarge the tracking area. Our proposed setup features only one single grating in addition to a regular microscope. The aberration correction, along with Poisson and background denoising, are incorporated in our deconvolution-based fully-automated algorithm, which requires no empirical parameter-tuning. In experiments, we achieve spatial resolutions of 0.35um (lateral) and 0.5um (axial), which are comparable to the resolution that can be achieved with confocal deconvolution microscopy. We demonstrate a 3D video of moving bacteria recorded at 25 frames per second using our proposed computational multifocal microscopy technique.

Multi-focus microscope (MFM) provides a way to obtain 3D information by simultaneously capturing multiple focal planes. The naive method for MFM reconstruction is to stack the sub-images with alignment. However, the resolution in the z-axis in this method is limited by the number of acquired focal planes. In this work we build on a recent reconstruction algorithm for MFM, using information from multiple frames to improve the reconstruction quality. We propose two multiple-frame MFM image reconstruction algorithms: batch and recursive approaches. In the batch approach, we take multiple MFM frames and jointly estimate the 3D image and the motion for each frame. In the recursive approach, we utilize the reconstructed image from the previous frame. Experimental results show that the proposed algorithms produce a sequence of 3D object reconstruction with high quality that enable reconstruction of dynamic extended objects.

We present a computer vision system that can transcribe the text on tiny printed labels stacked beneath pinned insects (as found in museum collections). The approach uses multiple views of each label because the labels are often occluded by the pin, the insect specimen, and other labels. Our approach handles occlusion and the extreme viewing angles required to image the stacked labels. Automated image analysis identifies the lines of text and then aligns and rectifies the images. Combining the aligned and rectified images from multiple viewpoints enables us to create a composite image that can be read using optical character recognition tools (OCR) to extract the text. We provide experimental demonstration using both museum specimens and experimental test labels.

Realizing both high temporal and spatial resolution across a large volume is a key challenge for 3D fluorescent imaging. Towards achieving this objective, we introduce an interferometric multifocus microscopy (iMFM) system, a combination of multifocus microscopy (MFM) with two opposing objective lenses. We show that the proposed iMFM is capable of simultaneously producing multiple focal plane interferometry that provides axial super-resolution and hence isotropic 3D resolution with a single exposure. We design and simulate the iMFM microscope by employing two special diffractive optical elements. The point spread function of this new iMFM microscope is simulated and the image formation model is given. For reconstruction, we use the Richardson-Lucy deconvolution algorithm with total variation regularization for 3D extended object recovery, and a maximum likelihood estimator (MLE) for single molecule tracking. A method for determining an initial axial position of the molecule is also proposed to improve the convergence of the MLE. We demonstrate both theoretically and numerically that isotropic 3D nanoscopic localization accuracy is achievable with an axial imaging range of 2um when tracking a fluorescent molecule in three dimensions and that the diffraction limited axial resolution can be improved by 3-4 times in the single shot wide-field 3D extended object recovery. We believe that iMFM will be a useful tool in 3D dynamic event imaging that requires both high temporal and spatial resolution.

This paper presents the design and prototyping of hardware and software to address the problem of rapid and reliable 3D digitization of very large collections of pinned insects. Using the collection at the Field Museum of Natural History (FMNH) as a use case, a pipeline to ingest the entire collection of 4.5 million specimens in circa 1-2 years imposes a few second limit on average processing time per specimen. We describe the design and implementation of multi-camera systems capable of rapidly capturing light field imagery for 3D reconstruction of label surfaces and specimen in single snapshots consistent with this time constraint. With imagery captured using these prototype multi-cameras we demonstrate methods under development for 3D reconstruction of pinned insect specimens and for processing text on label surfaces.

We describe a framework for relating available data on hazards to impact on infrastructure. It is designed to be used by simulation platforms that may focus on a single infrastructure layer or aim to study interactions between interdependent layers. The Hazard Impact Framework (HIF) provides a flexible scheme for capturing and maintaining data on the response of elements of critical infrastructures to natural and man-made hazards. HIF also provides interfaces that enable systemwide assessment of the impact that these hazards have on the assets that construct and connect national critical infrastructures. A key use of HIF involves providing initial configurations of systems under study that include primary damage assessment across the elements of that system. These configurations can then be fed to simulation platforms to determine the level of service remaining in the damaged system, cascading effects that follow from layer interdependencies, or dynamic effects in the compromised system and to provide stochastic evaluation of likely outcomes to an event.

Journal Article Integrated Dynamic 3D Imaging of Microbial Processes and Communities in Rhizosphere Environments: The Argonne Small Worlds Project Get access K M Kemner, K M Kemner Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar M Hereld, M Hereld Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar N Scherer, N Scherer University of Chicago, Department of Chemistry, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar A Selewa, A Selewa University of Chicago, Department of Chemistry, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar X Wang, X Wang University of Chicago, Department of Chemistry, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar I Gdor, I Gdor University of Chicago, Department of Chemistry, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar M Daddysman, M Daddysman University of Chicago, Department of Chemistry, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar J Jureller, J Jureller University of Chicago, Department of Chemistry, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar T Huynh, T Huynh University of Chicago, Department of Chemistry, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar O Cossairt, O Cossairt Northwestern University, Department of Engineering, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar ... Show more A Katsaggelos, A Katsaggelos Northwestern University, Department of Engineering, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar K He, K He Northwestern University, Department of Engineering, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar S Yoo, S Yoo Northwestern University, Department of Engineering, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar N Matsuda, N Matsuda Northwestern University, Department of Engineering, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar B Glick, B Glick University of Chicago, Department of Molecular Genetics and Cell Biology, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar P La Riviere, P La Riviere University of Chicago, Department of Molecular Genetics and Cell Biology, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar J Austin, J Austin University of Chicago, Department of Molecular Genetics and Cell Biology, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar K Day, K Day University of Chicago, Department of Molecular Genetics and Cell Biology, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar T Chandler, T Chandler University of Chicago, Department of Molecular Genetics and Cell Biology, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar S Papanikou, S Papanikou University of Chicago, Department of Molecular Genetics and Cell Biology, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar N Ferrier, N Ferrier Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar D Sholto-Douglas, D Sholto-Douglas Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar D Gursoy, D Gursoy Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar O Antipova, O Antipova Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar C Soriano, C Soriano Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar S O'Brien, S O'Brien University of Chicago, Department of Chemistry, Chicago, United States Search for other works by this author on: Oxford Academic Google Scholar R Wilton, R Wilton Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar A Ahrendt, A Ahrendt Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar M Asplund, M Asplund Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar S Zerbs, S Zerbs Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar P Noirot, P Noirot Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar C Atkins, C Atkins Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar G Babnigg, G Babnigg Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar J Johnson, J Johnson Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar S Shinde, S Shinde Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar P Korajczyk, P Korajczyk Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar M F Noirot M F Noirot Argonne National Laboratory, Argonne, United States Search for other works by this author on: Oxford Academic Google Scholar Microscopy and Microanalysis, Volume 23, Issue S1, 1 July 2017, Pages 340–341, https://doi.org/10.1017/S1431927617002380 Published: 04 August 2017

The authors present a graph-based computational framework that facilitates the construction, instantiation, and analysis of large-scale optimisation and simulation applications of coupled infrastructure networks. The framework integrates the optimisation modelling package PLASMO and the simulation package DMNetwork (built around PETSc). These tools use a common graph-based modelling abstraction that enables them to achieve compatibility between interfaces and data structures and facilitates the modular creation and exchange of component models. The authors also describe how to embed these tools within complex computational workflows using SWIFT, which is a tool that facilitates parallel execution of multiple simulation runs and management of input and output data. Finally, the authors discuss how to use these capabilities to target coupled natural gas and electricity systems.