ENRICO is a coupled application developed under the U.S. Department of Energy's Exascale Computing Project (ECP) targeting the modeling of advanced nuclear reactors. It couples radiation transport with heat and fluid simulation, including the high-fidelity, highresolution Monte-Carlo code Shift and the Computational fluid dynamics code NekRS. NekRS is a highly-performant open-source code for simulation of incompressible and low-Mach fluid flow, heat transfer, and combustion with a particular focus on turbulent flows in complex domains. It is based on rapidly convergent high-order spectral element discretizations that feature minimal numerical dissipation and dispersion. State-of-the-art multilevel preconditioners, efficient high-order time-splitting methods, and runtime-adaptive communication strategies are built on a fast OCCA-based kernel library, libParanumal, to provide scalability and portability across the spectrum of current and future high-performance computing platforms. On Frontier, Nek5000/RS has recently achieved an unprecedented milestone in breaching over 1 billion spectral elements and 350 billion degrees of freedom. Shift has demonstrated the capability to transport upwards of 1 billion particles per second in full core nuclear reactor simulations featuring complete temperature-dependent, continuous-energy physics on Frontier. Shift achieved a weak-scaling efficiency of 97.8% on 8192 nodes of Frontier and calculated 6 reactions in 214,896 fuel pin regions below 1% statistical error yielding first-of-a-kind resolution for a Monte Carlo transport application.

The development of NekRS, a GPU-oriented thermal-fluids simulation code based on the spectral element method (SEM) is described. For performance portability, the code is based on the open concurrent compute abstraction and leverages scalable developments in the SEM code Nek5000 and in libParanumal, which is a library of high-performance kernels for high-order discretizations and PDE-based miniapps. Critical performance sections of the Navier–Stokes time advancement are addressed. Performance results on several platforms are presented, including scaling to 27,648 V100s on OLCF Summit, for calculations of up to 60B grid points (240B degrees-of-freedom).

Nek5000/RS, a highly-performant open-source spectral element code, has recently achieved an unprecedented milestone in the simulation of nuclear reactors: the first full core computational fluid dynamics simulations of reactor cores, including pebble beds with 352,625 pebbles and 98M spectral elements (51 billion gridpoints), advanced in less than 0.25 seconds per Navier-Stokes timestep. The authors present performance and optimization considerations necessary to achieve this milestone when running on all of Summit. These optimizations led to a fourfold reduction in time-to-solution, making it possible to perform high-fidelity simulations of a single flow-through time in less than six hours for a full reactor core under prototypical conditions.

The Poisson pressure solve resulting from the spectral element discretization of the incompressible Navier-Stokes equation requires fast, robust, and scalable preconditioning. In the current work, a parallel scaling study of Chebyshevaccelerated Schwarz and Jacobi preconditioning schemes is presented, with special focus on GPU architectures, such as OLCF’s Summit. Convergence properties of the Chebyshevaccelerated schemes are compared with alternative methods, such as low-order preconditioners combined with algebraic multigrid. Performance and scalability results are presented for a variety of preconditioner and solver settings. The authors demonstrate that Chebyshev-accelerated-Schwarz methods provide a robust and effective smoothing strategy when using p-multigrid as a preconditioner in a Krylovsubspace projector. The variety of cases to be addressed, on a wide range of processor counts, suggests that performance can be enhanced by automated run-time selection of the preconditioner and associated parameters.

The solution to the Poisson equation arising from the spectral element discretization of the incompressible Navier-Stokes equation requires robust preconditioning strategies. One such strategy is multigrid. To realize the potential of multigrid methods, effective smoothing strategies are needed. Chebyshev polynomial smoothing proves to be an effective smoother. However, there are several improvements to be made, especially at the cost of symmetry. For the same cost per iteration, a full V-cycle with $k$ order Chebyshev polynomial smoothing may be substituted with a half V-cycle with order $2k$ Chebyshev polynomial smoothing, wherein the smoother is omitted on the up-leg of the V-cycle. The choice of omitting the post-smoother in favor of higher order Chebyshev pre-smoothing is shown to be advantageous in cases where the multigrid approximation property constant, $C$, is large. Results utilizing Lottes's fourth-kind Chebyshev polynomial smoother are shown. These methods demonstrate substantial improvement over the standard Chebyshev polynomial smoother. The authors demonstrate the effectiveness of this scheme in $p$-geometric multigrid, as well as a 2D model problem with finite differences.

Nek5000/RS is a highly-performant open-source spectral element code for simulation of incompressible and low-Mach fluid flow, heat transfer, and combustion with a particular focus on turbulent flows in complex domains. It is based on high-order discretizations that realize the same (or lower) cost per gridpoint as traditional low-order methods. State-of-the-art multilevel preconditioners, efficient high-order time-splitting methods, and runtime-adaptive communication strategies are built on a fast OCCA-based kernel library, libParanumal, to provide scalability and portability across the spectrum of current and future high-performance computing platforms. On Summit, Nek5000/RS has recently achieved an milestone in the simulation of nuclear reactors: the first full-core computational fluid dynamics simulations of reactor cores, including pebble beds with>350,000 pebbles and 98M elements advanced in less than 0.25 seconds per Navier-Stokes timestep. With carefully tuned algorithms, it is possible to simulate a single flow-through time for a full reactor core in less than six hours on all of Summit.

Motivation: Phylogenetic placement (i.e., the insertion of a sequence into a phylogenetic tree) is a basic step in several bioinformatics pipelines, including taxon identification in metagenomic analysis and large scale phylogeny estimation. The most accurate current method is pplacer, which attempts to optimize the placement using maximum likelihood, but it frequently fails on datasets where the phylogenetic tree has 5000 leaves. APPLES is the current most scalable method, and EPA-ng, although more scalable than pplacer and more accurate than APPLES, also fails on many 50,000-taxon trees. Here we describe pplacerDC, a divide-and-conquer approach that enables pplacer to be used when the phylogenetic tree is very large. Results: Our study shows that pplacerDC has excellent accuracy and scalability, matching pplacer where pplacer can run, improving accuracy compared to APPLES and EPA-ng, and is able to run on datasets with up to 100,000 sequences. Availability: The pplacerDC code is available on GitHub at https://github.com/kodingkoning/pplacerDC.

In this paper we describe the research and development activities in the Center for Efficient Exascale Discretization within the US Exascale Computing Project, targeting state-of-the-art high-order finite-element algorithms for high-order applications on GPU-accelerated platforms. We discuss the GPU developments in several components of the CEED software stack, including the libCEED, MAGMA, MFEM, libParanumal, and Nek projects. We report performance and capability improvements in several CEED-enabled applications on both NVIDIA and AMD GPU systems.

The Poisson pressure solve resulting from the spectral element discretization of the incompressible Navier-Stokes equation requires fast, robust, and scalable preconditioning. In the current work, a parallel scaling study of Chebyshev-accelerated Schwarz and Jacobi preconditioning schemes is presented, with special focus on GPU architectures, such as OLCF's Summit. Convergence properties of the Chebyshev-accelerated schemes are compared with alternative methods, such as low-order preconditioners combined with algebraic multigrid. Performance and scalability results are presented for a variety of preconditioner and solver settings. The authors demonstrate that Chebyshev-accelerated-Schwarz methods provide a robust and effective smoothing strategy when using $p$-multigrid as a preconditioner in a Krylov-subspace projector. At the same time, optimal preconditioning parameters can vary for different geometries, problem sizes, and processor counts. This variance motivates the development of an autotuner to optimize solver parameters on-line, during the course of production simulations.

Alzheimer’s Disease is characterized by the formation of amyloid beta (A[Formula: see text] fibril plaques in the brain. These fibrils can be probed by solid state NMR (ssNMR), which leads to an ensemble of configurations that are compatible with the NMR signals. Typically, only the lowest energy conformer is considered in computer simulations that probe the stability of fibrils and their binding with drug candidates. This restriction could produce data that are not physiologically relevant if the NMR entries differ significantly in stability or binding affinities. In order to study this effect, we have investigated the variance in stability between members of NMR ensembles. Our test cases are a patient-derived A[Formula: see text]-fibril model and two in vitro A[Formula: see text]-fibril models from a previous study we performed on comparative stability. The latter two models allow us also to compare different staggering patterns. We observe significant variations in molecular flexibility, compactness and secondary structure, suggesting that the full NMR ensemble must be considered for a physiologically relevant description of A[Formula: see text] fibrils.

Currently, no drugs exist that can prevent or reverse Alzheimer's disease, a neurodegenerative disease associated with the presence, in the brain, of plaques that are composed of β-amyloid (Aβ) peptides. Recent studies suggest that angiotensin-converting enzyme (ACE) inhibitors, a set of drugs used to treat hypertension, may inhibit amyloid formation in vitro. In the present study, we investigate through computer simulations the binding of ACE inhibitors to patient-derived Aβ fibrils and contrast it with that of ACE inhibitors binding to in vitro generated fibrils. The binding affinities of the ACE inhibitors are compared with that of Congo red, a dye that is used to identify amyloid structures and that is known to be a weak inhibitor of Aβ aggregation. We find that ACE inhibitors have a lower binding affinity to the patient-derived fibrils than to in vitro generated ones. For patient-derived fibrils, their binding affinities are even lower than that of Congo red. Our observations raise doubts on the hypothesis that these drugs inhibit fibril formation in Alzheimer patients by interacting directly with the amyloids.

The amyloid beta (Aβ) oligomers and fibrils that are found in neural tissues of patients suffering from Alzheimer's disease may either cause or contribute to the pathology of the disease. In vitro, these Aβ-aggregates are characterized by structural polymorphism. However, recent solid state NMR data of fibrils acquired post mortem from the brains of two Alzheimer's patients indicate presence of only a single, patient-specific structure. Using enhanced molecular dynamic simulations we investigate the factors that modulate the stability of Aβ-fibrils. We find characteristic differences in molecular flexibility, dynamics of interactions, and structural behavior between the brain-derived Aβ-fibril structure and in vitro models. These differences may help to explain the lack of polymorphism in fibrils collected from patient brains, and have to be taken into account when designing aggregation inhibitors and imaging agents for Alzheimer's disease.