Autonomous experimentation is an emerging area of research, primarily related to autonomous vehicles, scientific combinatorial discovery approaches in materials science and drug discovery, and iterative research loops of planning, experimentation, and analysis. However, autonomous approaches developed in these contexts are difficult to apply to high-dimensional mapping technologies, such as scanning hyperspectral imaging of biological systems, due to sample complexity and heterogeneity. We briefly cover the history of adaptive sampling algorithms and surrogate modeling in order to define autonomous adaptive data acquisition as an objective-based, flexible building block for future biological imaging experimentation driven by intelligent infrastructure. We subsequently summarize the recent implementations of autonomous adaptive data acquisition (AADA) for scanning hyperspectral imaging, assess how these address the difficulties of autonomous approaches in hyperspectral imaging, and highlight the AADA design variation from a goal-oriented perspective. Finally, we present a modular AADA architecture that embeds AADA-driven flexible building blocks to address the challenge of time resolution for high-dimensional scanning hyperspectral imaging of nonequilibrium dynamical systems. In our example research-driven experimental design case, we propose an AADA infrastructure for time-resolved, noninvasive, and label-free scanning hyperspectral imaging of living biological systems. This AADA infrastructure can accurately target the correct state of the system for experimental workflows that utilize subsequent expensive, high-information-content analytical techniques.

A Gaussian Process (GP) is a prominent mathematical framework for stochastic function approximation in science and engineering applications. This success is largely attributed to the GP's analytical tractability, robustness, non-parametric structure, and natural inclusion of uncertainty quantification. Unfortunately, the use of exact GPs is prohibitively expensive for large datasets due to their unfavorable numerical complexity of $O(N^3)$ in computation and $O(N^2)$ in storage. All existing methods addressing this issue utilize some form of approximation --- usually considering subsets of the full dataset or finding representative pseudo-points that render the covariance matrix well-structured and sparse. These approximate methods can lead to inaccuracies in function approximations and often limit the user's flexibility in designing expressive kernels. Instead of inducing sparsity via data-point geometry and structure, we propose to take advantage of naturally-occurring sparsity by allowing the kernel to discover --- instead of induce --- sparse structure. The premise of this paper is that GPs, in their most native form, are often naturally sparse, but commonly-used kernels do not allow us to exploit this sparsity. The core concept of exact, and at the same time sparse GPs relies on kernel definitions that provide enough flexibility to learn and encode not only non-zero but also zero covariances. This principle of ultra-flexible, compactly-supported, and non-stationary kernels, combined with HPC and constrained optimization, lets us scale exact GPs well beyond 5 million data points.

Scientific user facilities present a unique set of challenges for image processing due to the large volume of data generated from experiments and simulations. Furthermore, developing and implementing algorithms for real-time processing and analysis while correcting for any artifacts or distortions in images remains a complex task, given the computational requirements of the processing algorithms. In a collaborative effort across multiple Department of Energy national laboratories, the "MLExchange" project is focused on addressing these challenges. MLExchange is a Machine Learning framework deploying interactive web interfaces to enhance and accelerate data analysis. The platform allows users to easily upload, visualize, label, and train networks. The resulting models can be deployed on real data while both results and models could be shared with the scientists. The MLExchange web-based application for image segmentation allows for training, testing, and evaluating multiple machine learning models on hand-labeled tomography data. This environment provides users with an intuitive interface for segmenting images using a variety of machine learning algorithms and deep-learning neural networks. Additionally, these tools have the potential to overcome limitations in traditional image segmentation techniques, particularly for complex and low-contrast images.

Machine learning (ML) algorithms are showing a growing trend in helping the scientific communities across different disciplines and institutions to address large and diverse data problems. However, many available ML tools are programmatically demanding and computationally costly. The MLExchange project aims to build a collaborative platform equipped with enabling tools that allow scientists and facility users who do not have a profound ML background to use ML and computational resources in scientific discovery. At the high level, we are targeting a full user experience where managing and exchanging ML algorithms, workflows, and data are readily available through web applications. Since each component is an independent container, the whole platform or its individual service(s) can be easily deployed at servers of different scales, ranging from a personal device (laptop, smart phone, etc.) to high performance clusters (HPC) accessed (simultaneously) by many users. Thus, MLExchange renders flexible using scenarios-users could either access the services and resources from a remote server or run the whole platform or its individual service(s) within their local network.

Recent developments in 4th generation light sources and high-speed detectors are leading to rapid growth in data rates and data volumes, increasing the demand for automated data collection, handling/reduction/storage, and analysis processes.In combination with limited in-person access to experimental setups in times of the pandemic, portable and user-friendly tools for remote access as well as improved workflows are critical for enabling scientists from various disciplines to leverage ptychographic imaging to answer scientific questions.With the growing popularity of ptychography, a broad range of data formats, acquisition schemes, and algorithms has been developed over the years, e.g.[1][2][3].Whereas this variety has been advantageous to tackle different real-world deviations from the ideal ptychographic model such as partial incoherence [4], positioning errors [5], broad-bandwidth radiation [6], or multi-scattering [7], it also complicates the comparability and reproducibility of results.With ptychography being established as an everyday workhorse technique at many instruments around the world, it is important to find common ground and establish standards to support reliable algorithm and collaborative software development addressing the big data challenges of today and the future.In this presentation, I will cover recent cross-facility efforts [8] to develop and promote data standards for ptychography.Furthermore, I will give an overview of ongoing software development at the Advanced Light Source in collaboration with the other DOE light sources for building data acquisition and analysis tools leveraging existing python packages with an outlook for future progress in terms of remote access and workflows.

Mathematical optimization lies at the core of many science and industry applications. One important issue with many current optimization strategies is a well-known trade-off between the number of function evaluations and the probability to find the global, or at least sufficiently high-quality local optima. In machine learning (ML), and by extension in active learning - for instance for autonomous experimentation - mathematical optimization is often used to find the underlying uncertain surrogate model from which subsequent decisions are made and therefore ML relies on high-quality optima to obtain the most accurate models. Active learning often has the added complexity of missing offline training data; therefore, the training has to be conducted during the data collection which can stall the acquisition if standard methods are used. In this work, we highlight recent efforts to create a high-performance hybrid optimization algorithm (HGDL), combining derivative-free global optimization strategies with local, derivative-based optimization, ultimately yielding an ordered list of unique local optima. Redundancies are avoided by deflating the objective function around earlier encountered optima. HGDL is designed to take full advantage of parallelism by having the most computationally expensive process, the local first and second-order-derivative-based optimizations, run in parallel on separate compute nodes in separate processes. In addition, the algorithm runs asynchronously; as soon as the first solution is found, it can be used while the algorithm continues to find more solutions. We apply the proposed optimization and training strategy to Gaussian-Process-driven stochastic function approximation and active learning.

It has become increasingly common for researchers to utilize methods that identify weather features in climate models. There is an increasing recognition that the uncertainty associated with choice of detection method may affect our scientific understanding. For example, results from the Atmospheric River Tracking Method Intercomparison Project (ARTMIP) indicate that there are a broad range of plausible atmospheric river (AR) detectors and that scientific results can depend on the algorithm used. There are similar examples from the literature on extratropical cyclones and tropical cyclones. It is therefore imperative to develop detection techniques that explicitly quantify the uncertainty associated with the detection of events. We seek to answer the following question: given a “plausible” AR detector, how does uncertainty in the detector quantitatively impact scientific results? We develop a large dataset of global AR counts, manually identified by a set of eight researchers with expertise in atmospheric science, which we use to constrain parameters in a novel AR detection method. We use a Bayesian framework to sample from the set of AR detector parameters that yield AR counts similar to the expert database of AR counts; this yields a set of “plausible” AR detectors from which we can assess quantitative uncertainty. This probabilistic AR detector has been implemented in the Toolkit for Extreme Climate Analysis (TECA), which allows for efficient processing of petabyte-scale datasets. We apply the TECA Bayesian AR Detector, TECA-BARD v1.0.1, to the MERRA-2 reanalysis and show that the sign of the correlation between global AR count and El Niño–Southern Oscillation depends on the set of parameters used.

Great strides have been made to enable performance, portability, and productivity in HPC, but the focus has so far been on standalone applications and on-node programming models. Complex workflows comprising many communicating orchestrators, services, and simulations are redefining what modern HPC applications look like, and it is time to revisit what performance, portability, and performance mean when considered at the workflow level. What definitions make the most sense for workflows? How can workflows be deployed across HPC centers with incompatible software? And how can we ensure that not just one but many applications are deployed reliably and in an optimized way across different clusters? This paper highlights trends, reveals new challenges, and points to some potential solutions.

The constant stream of new users, samples, and experimental approaches at many light source beam lines means a single refined set of processing steps and parameters can seldom be re-used, and instead an interactive approach to exploring parameter space is required. But the many large data sets often generated during experiments mean that testing even one set of parameters is time intensive. We present two approaches to leverage parallel high performance computing to quickly and interactively explore parameter space for a large number of data sets, in the context of synchrotron tomography. The first approach uses a combination of template Jupyter notebooks with custom widgets, along with Papermill, and Dask. The second approach leverages the workflow infrastructure of Xi-Cam and Cam-Link, an ecosystem designed for real-time interactive exploration of data at user facilities.

The analysis of chemical states and morphology in nanomaterials is central to many areas of science. We address this need with an ultrahigh-resolution scanning transmission soft x-ray microscope. Our instrument provides multiple analysis tools in a compact assembly and can achieve few-nanometer spatial resolution and high chemical sensitivity via x-ray ptychography and conventional scanning microscopy. A novel scanning mechanism, coupled to advanced x-ray detectors, a high-brightness x-ray source, and high-performance computing for analysis provide a revolutionary step forward in terms of imaging speed and resolution. We present x-ray microscopy with 8-nm full-period spatial resolution and use this capability in conjunction with operando sample environments and cryogenic imaging, which are now routinely available. Our multimodal approach will find wide use across many fields of science and facilitate correlative analysis of materials with other types of probes.

Journal Article Towards a BES Light Source Wide Event-triggered Tomography Data Analysis Pipeline Using a Sustainable Software Stack Get access Harinarayan Krishnan, Harinarayan Krishnan Lawrence Berkeley National Laboratory, Richmond, California, United States Search for other works by this author on: Oxford Academic Google Scholar Tom Caswell, Tom Caswell Brookhaven National Lab, Upton, New York, United States Search for other works by this author on: Oxford Academic Google Scholar Daniel Allan Daniel Allan Brookhaven National Lab, Upton, New York, United States Search for other works by this author on: Oxford Academic Google Scholar Microscopy and Microanalysis, Volume 26, Issue S2, 1 August 2020, Pages 3092–3094, https://doi.org/10.1017/S143192762002379X Published: 01 August 2020

Noting a strong imperative to understand precipitation extremes, and that considerable uncertainty affects observational data sets, this paper compares the representation of extremes in a number of widely used daily gridded products, derived from rain gauge data, satellite retrieval and reanalysis for the conterminous United States. Analysis is based upon the concept of tail dependence arising in multivariate extreme value theory, and we infer the level of temporal dependence in the joint tail of the precipitation probability distribution for pairwise comparisons of products. In this way, we consider the range of products more like an ensemble and examine the relationships between members, and do not attempt to define, or compare products to, some ground truth. Linear correlation between products is also computed. Considerable discrepancy between groups of products, both annually and seasonally, is linked to source data and complex terrain. In particular, products based on rain gauge data showed remarkable similarity, but differed considerably, showing almost total loss of extremal dependence during DJF in mountainous regions, when compared with satellite products. Additionally, simulated re-forecasts revealed reasonable temporal agreement with large scale generated extremes. The diversity and extent of discrepancies identified across all products raises important questions about their use, and we urge caution, particularly for products derived from satellite data.

Atmospheric rivers (ARs) are now widely known for their association with high-impact weather events and long-term water supply in many regions. Researchers within the scientific community have developed numerous methods to identify and track of ARs-a necessary step for analyses on gridded data sets, and objective attribution of impacts to ARs. These different methods have been developed to answer specific research questions and hence use different criteria (e.g., geometry, threshold values of key variables, and time dependence). Furthermore, these methods are often employed using different reanalysis data sets, time periods, and regions of interest. The goal of the Atmospheric River Tracking Method Intercomparison Project (ARTMIP) is to understand and quantify uncertainties in AR science that arise due to differences in these methods. This paper presents results for key AR-related metrics based on 20+ different AR identification and tracking methods applied to Modern-Era Retrospective Analysis for Research and Applications Version 2 reanalysis data from January 1980 through June 2017. We show that AR frequency, duration, and seasonality exhibit a wide range of results, while the meridional distribution of these metrics along selected coastal (but not interior) transects are quite similar across methods. Furthermore, methods are grouped into criteria-based clusters, within which the range of results is reduced. AR case studies and an evaluation of individual method deviation from an all-method mean highlight advantages/disadvantages of certain approaches. For example, methods with less (more) restrictive criteria identify more (less) ARs and AR-related impacts. Finally, this paper concludes with a discussion and recommendations for those conducting AR-related research to consider.

There is a growing research interest in understanding extreme weather in the context of anthropogenic climate change, posing a requirement for new tailored climate data products. Here we introduce the Climate of the 20th Century Plus Detection and Attribution project (C20C + D&A), an international collaboration generating a product specifically intended for diagnosing causes of changes in extreme weather and for understanding uncertainties in that diagnosis. The project runs multiple dynamical models of the atmosphere-land system under observed historical conditions as well as under naturalised versions of those observed conditions, with the latter representing how the climate system might have evolved in the absence of anthropogenic interference. Each model generates large ensembles of simulations with different initial conditions for each historical scenario, providing a large sample size for understanding interannual variability, long-term trends, and the anthropogenic role in rare types of weather. This paper describes the C20C + D&A project design, implementation, strengths, and limitations, and also discusses various activities such as this special issue of Weather and Climate Extremes dedicated to "First results of the C20C + Detection and Attribution project".

The Atmospheric River Tracking Method Intercomparison Project (ARTMIP) is an international collaborative effort to understand and quantify the uncertainties in atmospheric river (AR) science based on detection algorithm alone. Currently, there are many AR identification and tracking algorithms in the literature with a wide range of techniques and conclusions. ARTMIP strives to provide the community with information on different methodologies and provide guidance on the most appropriate algorithm for a given science question or region of interest. All ARTMIP participants will implement their detection algorithms on a specified common dataset for a defined period of time. The project is divided into two phases: Tier 1 will utilize the Modern-Era Retrospective analysis for Research and Applications, version 2 (MERRA-2) reanalysis from January 1980 to June 2017 and will be used as a baseline for all subsequent comparisons. Participation in Tier 1 is required. Tier 2 will be optional and include sensitivity studies designed around specific science questions, such as reanalysis uncertainty and climate change. High-resolution reanalysis and/or model output will be used wherever possible. Proposed metrics include AR frequency, duration, intensity, and precipitation attributable to ARs. Here, we present the ARTMIP experimental design, timeline, project requirements, and a brief description of the variety of methodologies in the current literature. We also present results from our 1-month proof-of-concept trial run designed to illustrate the utility and feasibility of the ARTMIP project.

X-ray imaging allows biologists to retrieve the atomic arrangement of proteins and doctors the capability to view broken bones in full detail. In this context, ptychography has risen as a reference imaging technique. It provides resolutions of one billionth of a meter, macroscopic field of view, or the capability to retrieve chemical or magnetic contrast, among other features. The goal is to reconstruct a 2D visualization of a sample from a collection of diffraction patterns generated from the interaction of a light source with the sample. The data collected is typically two orders of magnitude bigger than the final image reconstructed, so high performance solutions are normally desired. One of the latest advances in ptychography imaging is the development of Ptycho-ADMM, a new ptychography reconstruction algorithm based on the Alternating Direction Method of Multipliers (ADMM). Ptycho-ADMM provides faster convergence speed and better quality reconstructions, all while being more resilient to noise in comparison with state-of-the-art methods. The downside of Ptycho-ADMM is that it requires additional computation and a larger memory footprint compared to simpler solutions. In this paper we tackle the computational requirements of Ptycho-ADMM, and design the first high performance multi-GPU solution of the method. We analyze and exploit the parallelism of Ptycho-ADMM to make use of multiple GPU devices. The proposed implementation achieves reconstruction times comparable to other GPU-accelerated high performance solutions, while providing the enhanced reconstruction quality of the Ptycho-ADMM method.

We examine the effect of the 20th and recent 21st century anthropogenic climate change on high temperature extremes as simulated by four global atmospheric general circulation models submitted to the Climate of the 20th Century Plus Detection and Attribution project. This coordinated experiment is based upon two large ensembles simulations for each participating model. The “world that was” simulations are externally forced as realistically as possible. The “world that might have been” is identical except that the influence of human forcing is removed but natural forcing agents and variations in ocean and sea ice are retained. We apply a stationary generalized extreme value analysis to the annual maxima of the three day average of the daily maximum surface air temperature, finding that long period return values have been increased by human activities between 1 and 3 C over most land areas. Corresponding changes in the probability of achieving long period non-industrial return values in the industrialized world are also presented. We find that most regions experience increases in the frequency and intensity of extremely hot three day periods, but anthropogenic sulfate aerosol forcing changes locally can decrease these measures of heat waves in some models.