ExaFEL is an HPC-capable X-ray Free Electron Laser (XFEL) data analysis software suite for both Serial Femtosecond Crystallography (SFX) and Single Particle Imaging (SPI) developed in collaboration with the Linac Coherent Lightsource (LCLS), Lawrence Berkeley National Laboratory (LBNL) and Los Alamos National Laboratory. ExaFEL supports real-time data analysis via a cross-facility workflow spanning LCLS and HPC centers such as NERSC and OLCF. Our work therefore constitutes initial path-finding for the US Department of Energy's (DOE) Integrated Research Infrastructure (IRI) program. We present the ExaFEL team's 7 years of experience in developing real-time XFEL data analysis software for the DOE's exascale supercomputers. We present our experiences and lessons learned with the Perlmutter and Frontier supercomputers. Furthermore we outline essential data center services (and the implications for institutional policy) required for real-time data analysis. Finally we summarize our software and performance engineering approaches and our experiences with NERSC's Perlmutter and OLCF's Frontier systems. This work is intended to be a practical blueprint for similar efforts in integrating exascale compute resources into other cross-facility workflows.

Advances in XFEL and detector technologies are on track to be able to enable diffraction experiments in the kHz and even MHz regimes at several linear accelerators worldwide in the coming years (Zastrau et al. 2021;Zhang et al. 2020).The computational demands of such experiments are daunting from many angles, but their scientific potential is equally incredible (Blaschke et al. 2021).One type of experiment piloted at current LCLS pulse repetition rates aims to pinpoint oxidation states of individual metal ions in a crystal structure without scanning the metal edge -or, more accurately, by collecting diffraction data from all wavelengths simultaneously and deconvoluting their contributions after the fact (Sauter et al. 2020).The natural bandwidth of LCLS pulses is already optimal for spanning the metal edge, and the stochastic individual spectra can be recorded in order to track their contributions at each wavelength.The diffraction signal attributable to each wavelength accumulates at a slightly different position on the detector for each Bragg peak, since each wavelength produces slightly different diffracting conditions, producing radially streaked reflections.The distribution of signal depends on several factors: the spectrum of the X-ray pulse irradiating the crystal, the precise crystal parameters and orientation, the wavelengthdependent parallax effect and the point-spread function on the detector.Selecting the values for each of these parameters that best fit the pixel-wise image data allows us to track and individually recover the contributions of each wavelength to each Bragg peak.This process has been validated on simulated data with the program nanoBragg (Mendez et al. 2020).To extend it to experimental data, we require the ability to refine individual pixel observations against the crystal model in a computationally feasible manner.The development of diffBragg addresses this step by computing first derivatives of complete (pixel-wise) images to use in global minimization.Finally, incorporation of this approach into the existing xfel.stills_processpipeline and the codevelopment of these tools with the exascale computing framework necessary to use them at scale are expected to make this approach tractable at upcoming XFEL experiments.

Inorganic–organic hybrid materials represent a large share of newly reported structures, owing to their simple synthetic routes and customizable properties 1 . This proliferation has led to a characterization bottleneck: many hybrid materials are obligate microcrystals with low symmetry and severe radiation sensitivity, interfering with the standard techniques of single-crystal X-ray diffraction 2,3 and electron microdiffraction 4–11 . Here we demonstrate small-molecule serial femtosecond X-ray crystallography (smSFX) for the determination of material crystal structures from microcrystals. We subjected microcrystalline suspensions to X-ray free-electron laser radiation 12,13 and obtained thousands of randomly oriented diffraction patterns. We determined unit cells by aggregating spot-finding results into high-resolution powder diffractograms. After indexing the sparse serial patterns by a graph theory approach 14 , the resulting datasets can be solved and refined using standard tools for single-crystal diffraction data 15–17 . We describe the ab initio structure solutions of mithrene (AgSePh) 18–20 , thiorene (AgSPh) and tethrene (AgTePh), of which the latter two were previously unknown structures. In thiorene, we identify a geometric change in the silver–silver bonding network that is linked to its divergent optoelectronic properties 20 . We demonstrate that smSFX can be applied as a general technique for structure determination of beam-sensitive microcrystalline materials at near-ambient temperature and pressure.

Light-driven oxidation of water to molecular oxygen is catalyzed by the oxygen-evolving complex (OEC) in Photosystem II (PS II). This multi-electron, multi-proton catalysis requires the transport of two water molecules to and four protons from the OEC. A high-resolution 1.89 Å structure obtained by averaging all the S states and refining the data of various time points during the S 2 to S 3 transition has provided better visualization of the potential pathways for substrate water insertion and proton release. Our results indicate that the O1 channel is the likely water intake pathway, and the Cl1 channel is the likely proton release pathway based on the structural rearrangements of water molecules and amino acid side chains along these channels. In particular in the Cl1 channel, we suggest that residue D1-E65 serves as a gate for proton transport by minimizing the back reaction. The results show that the water oxidation reaction at the OEC is well coordinated with the amino acid side chains and the H-bonding network over the entire length of the channels, which is essential in shuttling substrate waters and protons.

Most crystallographic data processing methods use pixel integration. In serial femtosecond crystallography (SFX), the intricate interaction between the reciprocal lattice point and the Ewald sphere is integrated out by averaging symmetrically equivalent observations recorded across a large number (10(4)-10(6)) of exposures. Although sufficient for generating biological insights, this approach converges slowly, and using it to accurately measure anomalous differences has proved difficult. This report presents a novel approach for increasing the accuracy of structure factors obtained from SFX data. A physical model describing all observed pixels is defined to a degree of complexity such that it can decouple the various contributions to the pixel intensities. Model dependencies include lattice orientation, unit-cell dimensions, mosaic structure, incident photon spectra and structure factor amplitudes. Maximum likelihood estimation is used to optimize all model parameters. The application of prior knowledge that structure factor amplitudes are positive quantities is included in the form of a reparameterization. The method is tested using a synthesized SFX dataset of ytterbium(III) lysozyme, where each X-ray laser pulse energy is centered at 9034 eV. This energy is 100 eV above the Yb3+ L-III absorption edge, so the anomalous difference signal is stable at 10 electrons despite the inherent energy jitter of each femtosecond X-ray laser pulse. This work demonstrates that this approach allows the determination of anomalous structure factors with very high accuracy while requiring an order-of-magnitude fewer shots than conventional integration-based methods would require to achieve similar results.

In oxygenic photosynthesis, light-driven oxidation of water to molecular oxygen is carried out by the oxygen-evolving complex (OEC) in photosystem II (PS II). Recently, we reported the room-temperature structures of PS II in the four (semi)stable S-states, S1, S2, S3, and S0, showing that a water molecule is inserted during the S2 → S3 transition, as a new bridging O(H)-ligand between Mn1 and Ca. To understand the sequence of events leading to the formation of this last stable intermediate state before O2 formation, we recorded diffraction and Mn X-ray emission spectroscopy (XES) data at several time points during the S2 → S3 transition. At the electron acceptor site, changes due to the two-electron redox chemistry at the quinones, QA and QB, are observed. At the donor site, tyrosine YZ and His190 H-bonded to it move by 50 µs after the second flash, and Glu189 moves away from Ca. This is followed by Mn1 and Mn4 moving apart, and the insertion of OX(H) at the open coordination site of Mn1. This water, possibly a ligand of Ca, could be supplied via a "water wheel"-like arrangement of five waters next to the OEC that is connected by a large channel to the bulk solvent. XES spectra show that Mn oxidation (τ of ∼350 µs) during the S2 → S3 transition mirrors the appearance of OX electron density. This indicates that the oxidation state change and the insertion of water as a bridging atom between Mn1 and Ca are highly correlated.

A nonlinear least-squares method for refining a parametric expression describing the estimated errors of reflection intensities in serial crystallographic (SX) data is presented. This approach, which is similar to that used in the rotation method of crystallographic data collection at synchrotrons, propagates error estimates from photon-counting statistics to the merged data. Here, it is demonstrated that the application of this approach to SX data provides better SAD phasing ability, enabling the autobuilding of a protein structure that had previously failed to be built. Estimating the error in the merged reflection intensities requires the understanding and propagation of all of the sources of error arising from the measurements. One type of error, which is well understood, is the counting error introduced when the detector counts X-ray photons. Thus, if other types of random errors (such as readout noise) as well as uncertainties in systematic corrections (such as from X-ray attenuation) are completely understood, they can be propagated along with the counting error, as appropriate. In practice, most software packages propagate as much error as they know how to model and then include error-adjustment terms that scale the error estimates until they explain the variance among the measurements. If this is performed carefully, then during SAD phasing likelihood-based approaches can make optimal use of these error estimates, increasing the chance of a successful structure solution. In serial crystallography, SAD phasing has remained challenging, with the few examples of de novo protein structure solution each requiring many thousands of diffraction patterns. Here, the effects of different methods of treating the error estimates are estimated and it is shown that using a parametric approach that includes terms proportional to the known experimental uncertainty, the reflection intensity and the squared reflection intensity to improve the error estimates can allow SAD phasing even from weak zinc anomalous signal.

The vast amount of data produced by XFEL experiments requires high performance computing for merging highly redundant measurements of Bragg intensities in real time. We have developed algorithms in the XFEL data processing package cctbx.xfel to distribute all steps of the merging process (data loading, filtering, scaling, postrefinement, error analysis, and averaging) between multiple compute nodes and CPU cores. This software, utilizing Message Passing Interface (MPI), has been tested on the Cray XC40 Cori supercomputer at the National Energy Research Supercomputing Center. We processed two data sets acquired at the Linac Coherent Light Source. A lysozyme data set comprising 1.5 x 10 7 intensity measurements was loaded and averaged using 32 Cori Knights Landing nodes (2,176 CPU cores) in less than 5 minutes. A Photosystem II dataset with 7.5 x 10 9 intensity measurements was processed using 82 nodes (5,576 cores) in 20 minutes.

In recent days, rewritable optical media DVD-RAM is widely used for recording massive data or movies.It is well-known that this recording is governed by a crystalline-amorphous transition of thin films, such as Ge 2 Sb 2 Te 5 .It was, however, difficult to understand the very fast recording and erasing mechanism by a normal idea of laser-induced melting and re-crystallizing process, and the real mechanism was mysterious.Recently, Kolobov et al. carried out an XAFS study on Ge 2 Sb 2 Te 5 thin film [1], and found that the crystal thin film does not possess a hexagonal structure as in the bulk, but more likely consists of highly distorted rocksalt structure.The laser-induced amorphization is concluded to be due to an umbrella-flip motion of Ge atoms from an octahedral position in crystal into a tetrahedral position.Since structural information obtained from XAFS measurements is limited to be a direction-averaged pair distribution function, another probe is necessary to confirm the above unique model.X-ray fluorescence holography (XFH) is a technique that allows one to obtain a three-dimensional image around a specific element [2,3].We have measured XFH on Ge 2 Sb 2 Te 5 single-crystal thin film at the beamline BL37XU of the SPring-8.The sample with about 2 microns thickness was placed on a two-axes table of a diffractometer installed at the beamline.The Ge Kα fluorescence X-rays were collected using an avalanche photodiode detector with a cylindrical graphitecrystal energy-analyzer.The XFH data were recorded at several incident energies of 22.0-27.0keV in steps of 0.5 keV.The measurements were performed by rotating two axes, 0°< φ < 360°in steps of about 0.3°and 0°< φ < 70°in steps of 0.5°, of the sample table, and detecting small intensity changes with angles.From the holographic patterns with the different incident energies, a three-dimensional atomic configuration image was constructed using Barton's algorithm.The obtained image clearly reveals that the Ge 2 Sb 2 Te 5 single-crystal thin film does not have a hexagonal structure around the Ge atoms.The detailed analysis is now in progress.

In recent days, rewritable optical media DVD-RAM is widely used for recording massive data or movies. It is well-known that this recording is governed by a crystalline-amorphous transition of thin films, such as Ge2Sb2Te5. It was, however, difficult to understand the very fast recording and erasing mechanism by a normal idea of laser-induced melting and re-crystallizing process, and the real mechanism was mysterious. Recently, Kolobov et al. carried out an XAFS study on Ge2Sb2Te5 thin film [1], and found that the crystal thin film does not possess a hexagonal structure as in the bulk, but more likely consists of highly distorted rocksalt structure. The laser-induced amorphization is concluded to be due to an umbrella-flip motion of Ge atoms from an octahedral position in crystal into a tetrahedral position. Since structural information obtained from XAFS measurements is limited to be a direction-averaged pair distribution function, another probe is necessary to confirm the above unique model. X-ray fluorescence holography (XFH) is a technique that allows one to obtain a three-dimensional image around a specific element [2,3]. We have measured XFH on Ge2Sb2Te5 single-crystal thin film at the beamline BL37XU of the SPring-8. The sample with about 2 microns thickness was placed on a two-axes table of a diffractometer installed at the beamline. The Ge Kα fluorescence X-rays were collected using an avalanche photodiode detector with a cylindrical graphitecrystal energy-analyzer. The XFH data were recorded at several incident energies of 22.0-27.0 keV in steps of 0.5 keV. The measurements were performed by rotating two axes, 0° < φ < 360° in steps of about 0.3° and 0° < φ < 70° in steps of 0.5°, of the sample table, and detecting small intensity changes with angles. From the holographic patterns with the different incident energies, a three-dimensional atomic configuration image was constructed using Barton’s algorithm. The obtained image clearly reveals that the Ge2Sb2Te5 single-crystal thin film does not have a hexagonal structure around the Ge atoms. The detailed analysis is now in progress.

X-raycrystallographers depend on software to process and analyze their data. These programswere often developed withemphasis on mathematical algorithms rather than inputand outputcontrol. As a result, these programs can be difficult to set up. This paperdiscusses a specific dataprocessing program, scaling, which is the laststep in crystallographydataprocessing. It computesthe scalefactors for the different X-rayimages and merges them to obtain a list ofthe Braggreflection intensities.

Most area detectors are normaly used with one or three circle goniometers.We have attached a Siemens CCD detector to our nonstandard four circle goniometer (oft'set Huber goniometer.Stoe xray optics.Siemens generator and a home-made LT-device and enclosure).For diagnostic purposes the CCD system can easily be replaced by the scintillation detector within an hour.Compared with a three circle system we can vary chi, but the Eulerian cradle resticts omega to a range of about 60 degrees.'vVe routinely collect a complete hemisphere of data.It appears to be better to collect as many equivalent reflections as possible, so we collect one structure per day rather than trying to work shifts to collect threee a day.The time for a data collection is thus independent of the unit-cell dimensions ... The quality of the data is significantly improved by the higher redundancy.This enables us to obtain high-angle data for extremly weakly diffracting crystals.Asemi-emphirical absorption conection can also be applied, exploiting the redundancy of the data.MS01.06.