Programming current supercomputers efficiently is a challenging task. Multiple levels of parallelism on the core, on the compute node, and between nodes need to be exploited to make full use of the system. Heterogeneous hardware architectures with accelerators further complicate the development process. waLBerla addresses these challenges by providing the user with highly efficient building blocks for developing simulations on block-structured grids. The block-structured domain partitioning is flexible enough to handle complex geometries, while the structured grid within each block allows for highly efficient implementations of stencil-based algorithms. We present several example applications realized with waLBerla, ranging from lattice Boltzmann methods to rigid particle simulations. Most importantly, these methods can be coupled together, enabling multiphysics simulations. The framework uses meta-programming techniques to generate highly efficient code for CPUs and GPUs from a symbolic method formulation.

Lattice Boltzmann methods are a popular mesoscopic alternative to macroscopic computational fluid dynamics solvers. Many variants have been developed that vary in complexity, accuracy, and computational cost. Extensions are available to simulate multi-phase, multi-component, turbulent, or non-Newtonian flows. In this work we present lbmpy, a code generation package that supports a wide variety of different methods and provides a generic development environment for new schemes as well. A high-level domain-specific language allows the user to formulate, extend and test various lattice Boltzmann schemes. The method specification is represented in a symbolic intermediate representation. Transformations that operate on this intermediate representation optimize and parallelize the method, yielding highly efficient lattice Boltzmann compute kernels not only for single- and two-relaxation-time schemes but also for multi-relaxation-time, cumulant, and entropically stabilized methods. An integration into the HPC framework waLBerla makes massively parallel, distributed simulations possible, which is demonstrated through scaling experiments on the SuperMUC-NG supercomputing system

A high-performance implementation of a multiphase lattice Boltzmann method based on the conservative Allen-Cahn model supporting high-density ratios and high Reynolds numbers is presented. Meta-programming techniques are used to generate optimized code for CPUs and GPUs automatically. The coupled model is specified in a high-level symbolic description and optimized through automatic transformations. The memory footprint of the resulting algorithm is reduced through the fusion of compute kernels. A roofline analysis demonstrates the excellent efficiency of the generated code on a single GPU. The resulting single GPU code has been integrated into the multiphysics framework waLBerla to run massively parallel simulations on large domains. Communication hiding and GPUDirect-enabled MPI yield near-perfect scaling behavior. Scaling experiments are conducted on the Piz Daint supercomputer with up to 2048 GPUs, simulating several hundred fully resolved bubbles. Further, validation of the implementation is shown in a physically relevant scenario—a three-dimensional rising air bubble in water.

Software development for applications in computational science and engineering has become complex in recent years. This is mainly due to the increasing parallelism and heterogeneity in modern computer architectures and to the more realistic physical and mathematical models that have to be processed. One idea to address this issue is to use code generation techniques. In contrast to manual implementations in a general-purpose computing language, they allow to integrate automatic code transforms to produce efficient code for different models and platforms. As an example the numerical solution of an elliptic partial differential equation via generated geometric multigrid solvers is considered. We present three code generation approaches for it and discuss their advantages and disadvantages with respect to performance, portability, and productivity.

We comment on the truncation error analysis and numerical artifacts of the D3Q19 lattice Boltzmann model reported in Silva et al. [3] . We present corrections for specific spatial truncation error terms in the momentum conservation equations. By introducing an improved discrete equilibrium for the D3Q19 stencil, we show that the reported spurious currents in a square channel duct flow are caused by the form of the discrete equilibrium and are not due to the structure and isotropy properties of the D3Q19 velocity set itself. Numerical experiments on a square channel and a more complex nozzle geometry confirm these results.

Lattice Boltzmann methods are a popular mesoscopic alternative to macroscopic computational fluid dynamics solvers. Many variants have been developed that vary in complexity, accuracy, and computational cost. Extensions are available to simulate multi-phase, multi-component, turbulent, or non-Newtonian flows. In this work we present lbmpy, a code generation package that supports a wide variety of different methods and provides a generic development environment for new schemes as well. A high-level domain-specific language allows the user to formulate, extend and test various lattice Boltzmann schemes. The method specification is represented in a symbolic intermediate representation. Transformations that operate on this intermediate representation optimize and parallelize the method, yielding highly efficient lattice Boltzmann compute kernels not only for single- and two-relaxation-time schemes but also for multi-relaxation-time, cumulant, and entropically stabilized methods. An integration into the HPC framework waLBerla makes massively parallel, distributed simulations possible, which is demonstrated through scaling experiments on the SuperMUC-NG supercomputing system

This article describes the development of automatic program generation technology to create scalable phase-field methods for material science applications. To simulate the formation of microstructures in metal alloys, we employ an advanced, thermodynamically consistent phase-field method. A state-of-the-art large-scale implementation of this model requires extensive, time-consuming, manual code optimization to achieve unprecedented fine mesh resolution. Our new approach starts with an abstract description based on free-energy functionals which is formally transformed into a continuous PDE and discretized automatically to obtain a stencil-based time-stepping scheme. Subsequently, an automatized performance engineering process generates highly optimized, performance-portable code for CPUs and GPUs. We demonstrate the efficiency for real-world simulations on large-scale GPU-based (PizDaint) and CPU-based (SuperMUC-NG) supercomputers. Our technique simplifies program development and optimization for a wide class of models. We further outperform existing, manually optimized implementations as our code can be generated specifically for each phase-field model and hardware configuration.

Clusters with GPUs are mainstream in HPC as shown by the last edition of the Top500 list, increasing the demand for GPU capable scientific computing software. Programming large scale GPU systems in an efficient and future-proof way present numerous challenges, such as optimizations for a variety of GPUs and interconnect hardware, hiding communication overhead with computation and efficient domain partitioning. We present an improvement to the CUDA-based communication of stencil applications in the WALBERLA framework, achieving scalability while supporting different GPUs and communication infrastructures. We utilize the lattice Boltzmann Method for fluid flows as a representative of stencil-based scientific computing and implement a communication hiding strategy that is capable of adjusting to a system's computing and communication capabilities. We compare the use of CUDAMemCopy with the use of customized pack/unpack kernels and show that packing achieves almost linear weak scaling behavior in the Santos Dumont supercomputer with up to 128 GPUs. We also show that the proposed approach is not sensitive to the direction of the domain partitioning, one of the biggest challenges when communicating 3D domains in GPU-based stencil simulations.

Realistic simulations in engineering or in the materials sciences can consume enormous computing resources and thus require the use of massively parallel supercomputers. The probability of a failure increases both with the runtime and with the number of system components. For future exascale systems, it is therefore considered critical that strategies are developed to make software resilient against failures. In this article, we present a scalable, distributed, diskless, and resilient checkpointing scheme that can create and recover snapshots of a partitioned simulation domain. We demonstrate the efficiency and scalability of the checkpoint strategy for simulations with up to 40 billion computational cells executing on more than 400 billion floating point values. A checkpoint creation is shown to require only a few seconds and the new checkpointing scheme scales almost perfectly up to more than 260, 000 (218) processes. To recover from a diskless checkpoint during runtime, we realize the recovery algorithms using ULFM MPI. The checkpointing mechanism is fully integrated in a state-of-the-art high-performance multi-physics simulation framework. We demonstrate the efficiency and robustness of the method with a realistic phase-field simulation originating in the material sciences and with a lattice Boltzmann method implementation.

Lattice Boltzmann simulations of three-dimensional, isothermal hydrodynamics often use either the D3Q19 or the D3Q27 velocity sets. While both models correctly approximate Navier-Stokes in the continuum limit, the D3Q19 model is computationally less expensive but has some known deficiencies regarding Galilean invariance, especially for high Reynolds number flows. In this work we present a novel methodology to construct lattice Boltzmann equilibria for hydrodynamics directly from the continuous Maxwellian equilibrium. While our new approach reproduces the well known LBM equilibrium for D2Q9 and D3Q27 lattice models, it yields a different equilibrium formulation for the D3Q19 stencil. This newly proposed formulation is shown to be more accurate than the widely used second order equilibrium, while having the same computation costs. We present a steady state Chapman-Enskog analysis of the standard and the improved D3Q19 model and conduct numerical experiments that demonstrate the superior accuracy of our newly developed D3Q19 equilibrium.

In spring 2015, the Leibniz Supercomputing Centre (Leibniz-Rechenzentrum, LRZ), installed their new Peta-Scale System SuperMUC Phase2. Selected users were invited for a 28 day extreme scale-out block operation during which they were allowed to use the full system for their applications. The following projects participated in the extreme scale-out workshop: BQCD (Quantum Physics), SeisSol (Geophysics, Seismics), GPI-2/GASPI (Toolkit for HPC), Seven-League Hydro (Astrophysics), ILBDC (Lattice Boltzmann CFD), Iphigenie (Molecular Dynamic), FLASH (Astrophysics), GADGET (Cosmological Dynamics), PSC (Plasma Physics), waLBerla (Lattice Boltzmann CFD), Musubi (Lattice Boltzmann CFD), Vertex3D (Stellar Astrophysics), CIAO (Combustion CFD), and LS1-Mardyn (Material Science). The projects were allowed to use the machine exclusively during the 28 day period, which corresponds to a total of 63.4 million core-hours, of which 43.8 million core-hours were used by the applications, resulting in a utilization of 69%. The top 3 users were using 15.2, 6.4, and 4.7 million core-hours, respectively.

The motion of ionic solutes and charged particles under the influence of an electric field and the ensuing hydrodynamic flow of the underlying solvent is ubiquitous in aqueous colloidal suspensions. The physics of such systems is described by a coupled set of differential equations, along with boundary conditions, collectively referred to as the electrokinetic equations. Capuani et al. [J. Chem. Phys. 121, 973 (2004)] introduced a lattice-based method for solving this system of equations, which builds upon the lattice Boltzmann algorithm for the simulation of hydrodynamic flow and exploits computational locality. However, thus far, a description of how to incorporate moving boundary conditions into the Capuani scheme has been lacking. Moving boundary conditions are needed to simulate multiple arbitrarily moving colloids. In this paper, we detail how to introduce such a particle coupling scheme, based on an analogue to the moving boundary method for the pure lattice Boltzmann solver. The key ingredients in our method are mass and charge conservation for the solute species and a partial-volume smoothing of the solute fluxes to minimize discretization artifacts. We demonstrate our algorithm’s effectiveness by simulating the electrophoresis of charged spheres in an external field; for a single sphere we compare to the equivalent electro-osmotic (co-moving) problem. Our method’s efficiency and ease of implementation should prove beneficial to future simulations of the dynamics in a wide range of complex nanoscopic and colloidal systems that were previously inaccessible to lattice-based continuum algorithms.

We present a Python extension to the massively parallel HPC simulation toolkit waLBerla. waLBerla is a framework for stencil based algorithms operating on block-structured grids, with the main application field being fluid simulations in complex geometries using the lattice Boltzmann method. Careful performance engineering results in excellent node performance and good scalability to over 400,000 cores. To increase the usability and flexibility of the framework, a Python interface was developed. Python extensions are used at all stages of the simulation pipeline: they simplify and automate scenario setup, evaluation, and plotting. We show how our Python interface outperforms the existing text-file-based configuration mechanism, providing features like automatic nondimensionalization of physical quantities and handling of complex parameter dependencies. Furthermore, Python is used to process and evaluate results while the simulation is running, leading to smaller output files and the possibility to adjust parameters dependent on the current simulation state. C++ data structures are exported such that a seamless interfacing to other numerical Python libraries is possible. The expressive power of Python and the performance of C++ make development of efficient code with low time effort possible.

A wide variety of growth patterns has been observed during the directional solidification of three-phase ternary eutectics in multi-component alloys. One hypothesized pattern is the spiral growth of two rod-like phases rotating around each other, which are embedded in a matrix phase. Possible evidence of spiral growth is found experimentally in longitudinal micrographs of the ternary eutectic system Ag–Al–Cu. The phase-field method allows the study of such spatially complex microstructures in order to gain deeper insights into the three-dimensional pattern formation of ternary eutectics. Based on the simulation parameters, which produced a high tilt angle in 2D lamellar growth, determined via systematic parameter studies, multiple spirals are detected in large-scale 3D simulations. Through the presented phase-field simulations, the possibility of the existence of spiral growth in ternary eutectics, reported from experiments, is confirmed.

Pattern formation in ternary eutectic microstructures is difficult to predict because of the complex interactions between diffusion and surface energies, as well as the large number of possible configurations which three phases can assume. The silver–aluminum–copper (Ag–Al–Cu) ternary eutectic is of particular interest to researchers, but its study is complicated by an unusually large solubility change. Due to this change, it is difficult to observe the patterns that form during the solid–liquid phase transition experimentally. This causes significant differences between the structure that forms during solidification and what is observed after traditional directional solidification processing. In order to model the solidification behavior and pattern formation in representative volume elements, large scale phase-field simulations are employed. Two different parameter sets are used. The first set uses the phase fractions and compositions of the as-solidified structure. The other used phase fractions found at lower temperatures in order to be comparable to experimental microstructures available in the literature. The second set of simulation results is compared quantitatively to experimental micrographs and found to be in good agreement. The predicted three-dimensional microstructures for the as-solidified structure are presented.

This study concentrates on finite-element-method (FEM)-based electroencephalography (EEG) forward simulation in which the electric potential evoked by neural activity in the brain is to be calculated at the surface of the head. The main advantage of the FEM is that it allows realistic modeling of tissue conductivity inhomogeneity. However, it is not straightforward to apply the classical model of a dipolar source with the FEM, due to its strong singularity and the resulting irregularity. The focus of this study is on comparing different methods to cope with this problem. In particular, we evaluate the accuracy of Whitney (Raviart-Thomas)-type dipole-like source currents compared to two reference dipole modeling methods: the St. Venant and partial integration approach. Common to all these methods is that they enable direct approximation of the potential field utilizing linear basis functions. In the present context, Whitney elements are particularly interesting, as they provide a simple means to model a divergence-conforming primary current vector field satisfying the square integrability condition. Our results show that a Whitney-type source model can provide simulation accuracy comparable to the present reference methods. It can lead to superior accuracy under optimized conditions with respect to both source location and orientation in a tetrahedral mesh. For random source orientations, the St. Venant approach turns out to be the method of choice over the interpolated version of the Whitney model. The overall moderate differences obtained suggest that practical aspects, such as the focality, should be prioritized when choosing a source model.

Microstructures forming during ternary eutectic directional solidification processes have significant influence on the macroscopic mechanical properties of metal alloys. For a realistic simulation, we use the well established thermodynamically consistent phase-field method and improve it with a new grand potential formulation to couple the concentration evolution. This extension is very compute intensive due to a temperature dependent diffusive concentration. We significantly extend previous simulations that have used simpler phase-field models or were performed on smaller domain sizes. The new method has been implemented within the massively parallel HPC framework waLBerla that is designed to exploit current supercomputers efficiently. We apply various optimization techniques, including buffering techniques, explicit SIMD kernel vectorization, and communication hiding. Simulations utilizing up to 262,144 cores have been run on three different supercomputing architectures and weak scalability results are shown. Additionally, a hierarchical, mesh-based data reduction strategy is developed to keep the I/O problem manageable at scale.

We present a Python extension to the massively parallel HPC framework WALBERLA. WALBERLA is a framework for stencil based algorithms operating on block-structured grids, with the main application field being fluid simulations in complex geometries using the lattice Boltzmann method. Careful performance engineering results in good scalability to over 400,000 cores. To increase the usability and flexibility of the framework, a Python interface was developed. Python extensions are used at all stages of the simulation pipeline: They simplify and automate scenario setup, evaluation, and plotting. We show how our Python interface outperforms the existing a text-filebased configuration mechanism, providing features like automatic nondimensionalization of physical quantities and handling of complex parameter dependencies. Furthermore, Python is used to process and evaluate results while the simulation is running, leading to smaller output files and the possibility to adjust parameters dependent on the current simulation state. C++ data structures are exported such that a seamless interfacing to other numerical Python libraries is possible. The expressive power of Python and the performance of C++ make development of efficient code with low time effort possible.

The adsorption process and the resulting dynamic surface tension in the context of protein foams were studied. A diffusion-advection equation is solved using a lattice Boltzmann method (LBM) in order to simulate the adsorption of surfactants on a surface. With different adsorption isotherms, different surfactants can be modelled. The advection is driven by a flow field coming from the LBM. The phase transition is implemented with a free surface LBM approach where the liquid-gas two-phase flow is simplified to a single-phase free surface flow by using a volume of fluid approach. Looking at the different time scales for diffusion and advection, which are determined by the diffusion coefficient and the viscosity, respectively, the LBM is limited due to time and space resolution. The rates of protein transport to a surface by diffusion and by advection are investigated which indicate that diffusion is only relevant for modelling long-time studies. For those time ranges and low concentrations, the diffusion of proteins from a bulk to a surface of a droplet is simulated and compared with the literature. As a next step, situations as in protein foams are assumed. High concentrations of proteins, e.g. as in milk, result in a simplified scenario where neither diffusion nor advection is important. This is analysed theoretically which suggests an instantaneous change of surface tension. To examine the stability of foam lamellae, this is used for further simulations. Two bubbles rise close to each other with globally different surface tensions as for pure water and water with proteins. Depending on these surface tensions and the initial distance, the bubbles coalesce faster for high surface tensions and show less secondary motions for lower surface tension. It is concluded that bubbles in protein foams coalesce only at shorter distances than in pure water.