Electromagnetic transient (EMT) simulation of power grids with high-fidelity models of inverter-based resources (IBRs) is time-consuming and difficult to scale. The necessity for high-fidelity models of IBRs that incorporate the dynamics of individual inverters within IBRs has been showcased in recent studies. These studies focused on events with partial power reduction in each IBR during a transmission line fault in the power grid. These types of events have been documented in multiple North American Electric Reliability Council (NERC) reports in the past decade. It is imperative then to find solutions to speed-up EMT simulations and scale the size of the region with IBRs studied in EMT simulations. In this paper, a combination of numerical simulation algorithms with high-performance computing techniques are employed in discretization and linear solvers employed in the proposed RE-INTEGRATE EMT simulation platform for power grid with IBRs. For ease of scalability, modular and object-oriented programming is used as these techniques are implemented. Additionally, automation software is developed to convert legacy software codes to the proposed RE-INTEGRATE EMT simulation platform. Thereafter, this platform is evaluated on multi-core central processing units (CPUs). Finally, scale-up tests are performed to showcase the scalability that is possible.

We discuss the challenges and propose research directions for using AI to revolutionize the development of high-performance computing (HPC) software. AI technologies, in particular large language models, have transformed every aspect of software development. For its part, HPC software is recognized as a highly specialized scientific field of its own. We discuss the challenges associated with leveraging state-of-the-art AI technologies to develop such a unique and niche class of software and outline our research directions in the two US Department of Energy–funded projects for advancing HPC Software via AI: Ellora and Durban.

Sunny is a Julia package designed to serve the needs of the quantum magnetism community. It supports the specification of a very broad class of spin models and a diverse suite of numerical solvers. These include powerful methods for simulating spin dynamics both in and out of equilibrium. Uniquely, it features a broad generalization of classical and semiclassical approaches to SU(N) coherent states, which is useful for studying systems exhibiting strong spin-orbit coupling or local entanglement effects. Sunny also offers a well-developed framework for calculating the dynamical spin structure factor, enabling direct comparison with scattering experiments. Ease of use is a priority, with tools for symmetry-guided modeling and interactive visualization.

We provide an overview of the software engineering efforts and their impact in QMCPACK, a production-level ab-initio Quantum Monte Carlo open-source code targeting high-performance computing (HPC) systems. Aspects included are: (i) strategic expansion of continuous integration (CI) targeting CPUs, using GitHub Actions own runners, and NVIDIA and AMD GPUs used in pre-exascale systems, (ii) incremental reduction of memory leaks using sanitizers, (iii) incorporation of Docker containers for CI and reproducibility, and (iv) refactoring efforts to improve maintainability, testing coverage, and memory lifetime management. We quantify the value of these improvements by providing metrics to illustrate the shift towards a predictive, rather than reactive, maintenance approach. Our goal, in documenting the impact of these efforts on QMCPACK, is to contribute to the body of knowledge on the importance of research software engineering (RSE) for the stewardship and advancement of community HPC codes to enable scientific discovery at scale.

We explore the development of a performance-portable CPU/GPU ecosystem to integrate two of the US Department of Energy’s (DOE’s) largest scientific instruments, the Oak Ridge Leadership Computing facility and the Spallation Neutron Source (SNS), both of which are housed at Oak Ridge National Laboratory. We select a relevant data reduction workflow use-case to obtain the differential scattering cross-section from data collected by SNS’s CORELLI and TOPAZ instruments. We compare the current CPU-only production implementation using the Garnet Python multiprocess package based on the Mantid C++ framework against our proposed CPU/GPU implementation that uses the LLVM-based, just-in-time Julia scientific language and the JACC.jl performance-portable package. Two proxy apps were developed: (i) an app for extracting relevant Mantid kernels (MDNorm) in C++ and (ii) the Julia MiniVATES.jl miniapp. We present performance results for NVIDIA A100 and AMD MI100 GPUs and AMD EPYC 7513 and 7662 CPUs. The results provide insights for future generations of data reduction software that can embrace performance portability for an integrated research infrastructure across DOE’s experimental and computational facilities.

Many areas of science exhibit physical processes that are described by high dimensional partial differential equations (PDEs), e.g., the 4D (Dorf et al., 2013), 5D (Candy et al., 2009) and 6D models (Juno et al., 2018) describing magnetized fusion plasmas, models describing quantum chemistry, or derivatives pricing (Bandrauk et al., 2007).Such problems are affected by the so-called "curse of dimensionality" where the number of degrees of freedom (or unknowns) required to be solved for scales as where is the number of grid points in any given dimension .A simple, albeit naive, 6D example is demonstrated in the left panel of Figure

Sparse-grid methods have recently gained interest in reducing the computational cost of solving high-dimensional kinetic equations. In this paper, we construct adaptive and hybrid sparse-grid methods for the Vlasov-Poisson-Lenard-Bernstein (VPLB) model. This model has applications to plasma physics and is simulated in two reduced geometries: a 0x3v space homogeneous geometry and a 1x3v slab geometry. We use the discontinuous Galerkin (DG) method as a base discretization due to its high-order accuracy and ability to preserve important structural properties of partial differential equations. We utilize a multiwavelet basis expansion to determine the sparse-grid basis and the adaptive mesh criteria. We analyze the proposed sparse-grid methods on a suite of three test problems by computing the savings afforded by sparse-grids in comparison to standard solutions of the DG method. The results are obtained using the adaptive sparse-grid discretization library ASGarD.

As computational capabilities improve, digital twins are becoming vital for evaluating equipment realistically in laboratories. This paper outlines a digital twin architecture for the power grid, employing electromagnetic transient (EMT) simulation alongside real-time simulation of power hardware and hierarchical control systems. EMT simulation occurs on a high-performance computing server for scalability. Additionally, the paper describes a workflow and real-time data streaming software facilitating connectivity among EMT simulation, hierarchical control systems, and power hardware. This software enables automated equipment connectivity in the laboratory for realistic evaluations, aiding in identifying necessary upgrades for both equipment control systems and the power grid.

Faults in the transmission grid have led to reduced power generation from power electronics resources that are typically not connected to the faulted transmission line. In many of the cases, partial loss of power is observed within the power electronics resources like large photovoltaic (PV) power plants. This phenomena is not captured in existing simulation models and/or simulators. High-fidelity switched system electromagnetic transient (EMT) dynamic models of PV power plants can improve the fidelity of models available for accurate analysis of the impact on PV plants during simulation of faults. However, these models are extremely computationally expensive and take a long time to simulate. Long simulation times limit the ability to use these models as larger regions are studied in EMT simulations with more power electronics resources. In this paper, numerical simulation algorithms are combined with high-performance computing techniques and applied to the high-fidelity switched system EMT model of PV plants. Using these techniques, a speed-up of up to 58x is obtained, while preserving the accuracy of the simulation at greater than 98%.

We provide an overview of the software engineering efforts and their impact in QMCPACK, a production-level ab-initio Quantum Monte Carlo open-source code targeting high-performance computing (HPC) systems. Aspects included are: (i) strategic expansion of continuous integration (CI) targeting CPUs, using GitHub Actions runners, and NVIDIA and AMD GPUs in pre-exascale systems, using self-hosted hardware; (ii) incremental reduction of memory leaks using sanitizers, (iii) incorporation of Docker containers for CI and reproducibility, and (iv) refactoring efforts to improve maintainability, testing coverage, and memory lifetime management. We quantify the value of these improvements by providing metrics to illustrate the shift towards a predictive, rather than reactive, sustainable maintenance approach. Our goal, in documenting the impact of these efforts on QMCPACK, is to contribute to the body of knowledge on the importance of research software engineering (RSE) for the sustainability of community HPC codes and scientific discovery at scale.

Data reduction is a critical step in a small-angle neutron scattering experiment. It corrects the data for instrument-specific artefacts, making it ready for analysis, interpretation, as well as for comparison against data collected with different small-angle neutron scattering instruments. Here, the drtsans software package developed at Oak Ridge National Laboratory for data reduction for the EQ-SANS, GP-SANS, and Bio-SANS instruments, which are located at the Spallation Neutron Source and High Flux Isotope Reactor, is described. The software and the rigorous development methods employed have positively impacted the scientific programs on the three SANS instruments.

Event-mode data collection presents remarkable new opportunities for time-of-flight neutron scattering studies of collective excitations, diffuse scattering from short-range atomic and magnetic structures, and neutron crystallography. In these experiments, large volumes of the reciprocal space are surveyed, often using different wavelengths and counting times. These data then have to be added together, with accurate propagation of the counting errors. This paper presents a statistically correct way of adding and histogramming the data for single-crystal time-of-flight neutron scattering measurements. In order to gain a broader community acceptance, particular attention is given to improving the efficiency of calculations.

General-purpose library application programming interfaces (APIs) for self-describing hierarchical scientific data storage, such as the HDF5 and NetCDF libraries, are traditionally of runtime nature. Runtime errors for entry existence and data types are typically caught later in the development process of higher-level application-specific APIs. In this paper, we propose exploiting modern C++ metaprogramming features to add compile-time type-safety to improve the interaction with a well-defined metadata-rich scientific schema in domain-specific hierarchical datasets. We tackle two aspects of common use: (i) direct data access, (ii) flexible “in-memory” index models for efficient search and data processing. The proposed APIs use C++17’s template type auto deduction features, C++11’s enum class for type-safety and C-style preprocessor macros for generative templated code. We showcase the pros and cons of our initial work on the standard NeXus schema used for annotating and storing experimental neutron scattering data at several facilities around the world on top of HDF5. Extendable compile-time type-safe APIs are a desirable feature that could be indexed by any modern integrated development environment (IDE). Hence, such APIs can help ease the learning curve for domain scientists using a less error-prone software interaction to enhance the findability of their data without resorting to a domain-specific language (DSL).

We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This makes it a scalability challenge, as the number of experiments stored increases in a single ensemble file. The present work follows up on our previous efforts on data management algorithms, to address identified input output (I/O) bottlenecks in Mantid, an open-source data analysis framework used across several neutron science facilities around the world. We reuse an in-memory binary-tree metadata index that resembles data access patterns, to provide a scalable search and extraction mechanism. In addition, several memory operations are refactored and optimized for the current common use cases, ranging most frequently from 10 to 180, and up to 360 separate measurement configurations. Results from this work show consistent speed ups in wall-clock time on the Mantid LoadMD routine, ranging from 19% to 23% on average, on ORNL production computing systems. The latter depends on the complexity of the targeted instrument-specific data and the system I/O and compute variability for the shared computational resources available to users of ORNL's Spallation Neutron Source (SNS) and the High Flux Isotope Reactor (HFIR) instruments. Nevertheless, we continue to highlight the need for more research to address reduction challenges as experimental data volumes, user time and processing costs increase.

Despite decades of research, characterization of the effects of polymer chain dispersity on the structural properties of block copolymer thin films remains challenging. We present an integrated experimental and modeling approach to characterize the morphology of thin films containing asymmetric diblock copolymers. Specifically, we used synergistic neutron reflectivity (NR) and self-consistent field theory (SCFT)-based modeling to realize unexpected morphology of thin films containing asymmetric copolymers. Using NR, a highly stable and reproducible mixed phase of coexisting cylinders and lamellar domains was discovered in asymmetric poly(deuterated-styrene-b-n butyl methacrylate) (dPS-PBMA) copolymer thin films containing 34% volume fraction of dPS. SCFT reveals how to obtain such a thermodynamically stable morphology in the presence of disperse majority block and asymmetric interactions of polymer species with surfaces. Stabilization of the coexisting domains is a consequence of the depth segregation based on chain-length distribution. The asymmetric chains microphase-separate into cylindrical domains close to the substrate and near-symmetric chains form lamellar domains at the air interface. In the absence of dispersity, the coexistence of cylindrical and lamellar domains is thermodynamically unstable because of the absence of depth segregation. Overall, such an effect of dispersity on diblock copolymer thin-film morphology reveals a unique and powerful strategy to create coexisting nanoscale domains and tailor properties of thin films.

Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.

Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL’s world-class high performance computing (HPC) facilities, and (2) our short-term efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows. We propose a more disruptive domain-specific solution: the No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO’s adoption by HFIR and SNS stakeholders.

TbBaFe2O5+δ (TBFO) is a mixed valence compound with an antiferromagnetic order (TN=450K) that changes along with the crystal structure and conductivity upon cooling below a Verwey-like transition (TV=280K). This type of double-cell-layered perovskite is useful to study putative charge ordering in fractional valence systems because the crystal structure develops unique sites for the different valence states. While the evolution of the static magnetic structure in the different charge ordered states of TBFO is known, in this study we investigate the effect of charge ordering on the magnetic interactions. Dispersive magnetic excitations measured using single crystal inelastic neutron scattering experiments at T=4KTV are modeled with linear-spin-wave theory to extract anisotropy and superexchange parameters. Below TV, we observe a sizable magnon gap due to an easy-axis magnetic anisotropy that closes above TV due to the evolution of easy-plane anisotropy. The magnetic interactions are found to be highly threedimensional and change across TV. Above TV, where the conductivity is greater, a damping term is required to model the spin waves that suggests a coupling to valence fluctuations. Disciplines Condensed Matter Physics Authors Daniel M. Pajerowski, Daniel K. Pratt, Steven E. Hahn, Wei Tian, Garrett E. Granroth, Alexander I. Kolesnikov, Alexey A. Taskin, Yoichi Ando, and Robert J. McQueeney This article is available at Iowa State University Digital Repository: https://lib.dr.iastate.edu/ameslab_manuscripts/ 603 PHYSICAL REVIEW B 101, 064418 (2020) Spin waves above and below the Verwey transition in TbBaFe2O5 Daniel M. Pajerowski ,1,* Daniel K. Pratt,2 Steven E. Hahn,1 Wei Tian,1 Garrett E. Granroth ,1 Alexander I. Kolesnikov ,1 Alexey A. Taskin ,3 Yoichi Ando,3 and Robert J. McQueeney4 1Neutron Scattering Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831, USA 2NIST Center for Neutron Research, National Institute of Standards and Technology, Gaithersburg, Maryland 20899, USA 3Physics Institute II, University of Cologne, 50937 Koeln, Germany 4Ames Laboratory, US DOE, Iowa State University, Ames, Iowa 50011, USA (Received 26 November 2019; revised manuscript received 27 January 2020; accepted 31 January 2020; published 18 February 2020) TbBaFe2O5+δ (TBFO) is a mixed valence compound with an antiferromagnetic order (TN = 450 K) that changes along with the crystal structure and conductivity upon cooling below a Verwey-like transition (TV = 280 K). This type of double-cell-layered perovskite is useful to study putative charge ordering in fractional valence systems because the crystal structure develops unique sites for the different valence states. While the evolution of the static magnetic structure in the different charge ordered states of TBFO is known, in this study we investigate the effect of charge ordering on the magnetic interactions. Dispersive magnetic excitations measured using single crystal inelastic neutron scattering experiments at T = 4 K < TV and T = 320 K > TV are modeled with linear-spin-wave theory to extract anisotropy and superexchange parameters. Below TV , we observe a sizable magnon gap due to an easy-axis magnetic anisotropy that closes above TV due to the evolution of easy-plane anisotropy. The magnetic interactions are found to be highly three-dimensional and change across TV . Above TV , where the conductivity is greater, a damping term is required to model the spin waves that suggests a coupling to valence fluctuations. DOI: 10.1103/PhysRevB.101.064418