Vela is a cloud-native system designed for LLM training workloads built using off-the-shelf hardware, Linux KVM-based virtualization, and a virtualized RDMA over Converged Ethernet (RoCE) network. Vela virtual machines (VMs) support peer-to-peer DMA between the GPUs and SRIOV-based network interface. In this paper, we share Vela's key architectural aspects with details from an NVIDIA A100 GPU-based deployment in one of the IBM Cloud data centers. Throughout the paper, we share insights and experiences from designing, building, and operating the system over a similar to 2.5 year timeframe to highlight the capabilities of readily available software and hardware technologies and the improvement opportunities for future AI systems, thereby making AI infrastructure more accessible to a broader community. As we evaluated the system for performance at similar to 1500 GPU scale, we achieved similar to 80% of the ideal throughput while training a 50 billion parameter decoder model using model parallelism, and similar to 70% per GPU FLOPS compared to a single VM with the High-Performance Linpack benchmark.

AI Infrastructure plays a key role in the speed and cost-competitiveness of developing and deploying advanced AI models. The current demand for powerful AI infrastructure for model training is driven by the emergence of generative AI and foundational models, where on occasion thousands of GPUs must cooperate on a single training job for the model to be trained in a reasonable time. Delivering efficient and high-performing AI training requires an end-to-end solution that combines hardware, software and holistic telemetry to cater for multiple types of AI workloads. In this report, we describe IBM's hybrid cloud infrastructure that powers our generative AI model development. This infrastructure includes (1) Vela: an AI-optimized supercomputing capability directly integrated into the IBM Cloud, delivering scalable, dynamic, multi-tenant and geographically distributed infrastructure for large-scale model training and other AI workflow steps and (2) Blue Vela: a large-scale, purpose-built, on-premises hosting environment that is optimized to support our largest and most ambitious AI model training tasks. Vela provides IBM with the dual benefit of high performance for internal use along with the flexibility to adapt to an evolving commercial landscape. Blue Vela provides us with the benefits of rapid development of our largest and most ambitious models, as well as future-proofing against the evolving model landscape in the industry. Taken together, they provide IBM with the ability to rapidly innovate in the development of both AI models and commercial offerings.

Public clouds are rapidly moving to support Non-Volatile Memory Express (NVMe) based storage to meet the ever-increasing I/O throughput and latency demands of modern workloads. They provide NVMe storage through virtual machines (VMs) where multiple VMs running on a host may share a physical NVMe device. The virtualization method used to share the NVMe capability has important performance, usability and security implications. In this paper, we propose three NVMe storage virtualization methods: PCI device passthrough, virtual block device method, and Storage Performance Development Kit (SPDK) virtual host target method. We evaluate these virtualization methods in terms of performance, scalability, CPU overhead, technology maturity, security, and availability to use one or more of these methods in IBM public cloud.

A new model for electron temperature gradient (ETG) modes is developed as a component of the multi-mode anomalous transport module [Rafiq et al., Phys Plasmas 20, 032506 (2013)] to predict a time-dependent electron temperature profile in conventional and low aspect ratio tokamaks. This model is based on two-fluid equations that govern the dynamics of low-frequency short- and long-wavelength electromagnetic toroidal ETG driven drift modes. A low collisionality NSTX discharge is used to scan the plasma parameter dependence on the ETG real frequency, growth rate, and electron thermal diffusivity. Electron thermal transport is discovered in the deep core region where modes are more electromagnetic in nature. Several previously reported gyrokinetic trends are reproduced, including the dependencies of density gradients, magnetic shear, β and gradient of β (β′), collisionality, safety factor, and toroidicity, where β is the ratio of the plasma pressure to the magnetic pressure. The electron heat diffusivity associated with the ETG mode is discovered to be on a scale consistent with the experimental diffusivity determined by power balance analysis.

Microtearing mode (MTM) real frequency, growth rate, magnetic fluctuation amplitude, and resulting electron thermal transport are studied in systematic NSTX scans of relevant plasma parameters. The dependency of the MTM real frequency and growth rate on plasma parameters, suitable for low and high collision NSTX discharges, is obtained by using the reduced MTM transport model [T. Rafiq et al., Phys. Plasmas 23, 062507 (2016)]. The plasma parameter dependencies are compared and found to be consistent with the results obtained from MTM using the gyrokinetic GYRO code. The scaling trend of collision frequency and plasma beta is found to be consistent with the global energy confinement trend observed in the NSTX experiment. The strength of the magnetic fluctuation is found to be consistent with the gyrokinetic estimate. In earlier studies, it was found that the version of the multi-mode (MM) anomalous transport model, which did not contain the effect of MTMs, provided an appropriate description of the electron temperature profiles in standard tokamak discharges and not in spherical tokamaks. When the MM model, which involves transport associated with MTMs, is incorporated in the TRANSP code and is used in the study of electron thermal transport in NSTX discharges, it is observed that the agreement with the experimental electron temperature profile is substantially improved.

High-performance computing (HPC) increasingly relies on heterogeneous architectures to achieve higher performance. In the Oak Ridge Leadership Facility (OLCF), Oak Ridge, TN, USA, this trend continues as its latest supercomputer, Summit, entered production in early 2019. The combination of IBM POWERS CPU and NVIDIA V100 GPU, along with a fast NVLink2 interconnect and other latest technologies, pushes system performance to a new height and breaks the exascale barrier by certain measures. Due to Summit powerful GPUs and much higher GPU-CPU ratio, offloading to accelerators becomes a requirement for any application, which intends to effectively use the system. To facilitate navigating a complex landscape of competing heterogeneous architectures, a collection of applications from a wide spectrum of scientic domains is selected for early adoption on Summit. In this article, the experience and lessons learned are summarized, in the hope of providing useful guidance to address new programming challenges, such as scalability, performance portability, and software maintainability, for future application development efforts on heterogeneous HPC systems.

Model order reduction is a modeling method that can facilitate the analysis and the control synthesis of distributed parameter systems, which are governed by partial differential equations. Many techniques have been proposed to further improve the property of the reduced order model based on full state information, which is usually not available in practical applications. In this paper, a reduced-order modeling method based on sparse point measurements is proposed for the benchmark system of flow past a cylinder. This method involves solving the dynamic optimization problems online, whose objective functions measure the differences between the predicted and observed variables on each time horizon. In this method, the sensor placement is also discussed, and we determine the sensor locations depending on a model-free optimization technique. Several numerical examples of different scenarios are simulated not only to illustrate the effectiveness of the proposed method but also to verify the discussion on the sensor placement strategy.

Mesoscopic simulations of hydrocarbon flow in source shales are challenging, in part due to the heterogeneous shale pores with sizes ranging from a few nanometers to a few micrometers. Additionally, the sub-continuum fluid-fluid and fluid-solid interactions in nano- to micro-scale shale pores, which are physically and chemically sophisticated, must be captured. To address those challenges, we present a GPU-accelerated package for simulation of flow in nano- to micro-pore networks with a many-body dissipative particle dynamics (mDPD) mesoscale model. Based on a fully distributed parallel paradigm, the code offloads all intensive workloads on GPUs. Other advancements, such as smart particle packing and no-slip boundary condition in complex pore geometries, are also implemented for the construction and the simulation of the realistic shale pores from 3D nanometer-resolution stack images. Our code is validated for accuracy and compared against the CPU counterpart for speedup. In our benchmark tests, the code delivers nearly perfect strong scaling and weak scaling (with up to 512 million particles) on up to 512 K2OX GPUs on Oak Ridge National Laboratory's (ORNL) Titan supercomputer. Moreover, a single-GPU benchmark on ORNL's SummitDev and IBM's AC922 suggests that the host-to-device NVLink can boost performance over PCIe by a remarkable 40%. Lastly, we demonstrate, through a flow simulation in realistic shale pores, that the CPU counterpart requires 840 Power9 cores to rival the performance delivered by our package with four V100 GPUs on ORNL's Summit architecture. This simulation package enables quick-turnaround and high-throughput mesoscopic numerical simulations for investigating complex flow phenomena in nano- to micro-porous rocks with realistic pore geometries. Program summary Program title: usERMESO 2. 5 Program files doi: http://dx.doi.org/10.17632/zzpv74bz9m.1 Licensing provisions: GNU General Public License 3 Programming language: CUDA C/C++ with MPI and OpenMP Nature of problem: Particle-based simulation of multiphase flow and fluid-solid interaction in nano to micro-scale pore networks of arbitrary pore geometries. Solution method: Fluid particles and solid wall particles are modeled with a many-body dissipative particle dynamics (mDPD) model- a mesoscopic model for coarse-grained fluid and solid molecules. The pore surface wall boundary for arbitrary surface geometries is modeled with a no-slip boundary condition for fluid particles that prevents fluid particles from indefinitely penetrating in the walls. The time evolution of the system is integrated using the Velocity-Verlet algorithm. Restrictions: The code is compatible with NVIDIA GPUs with compute capability 3.0 and above. Unusual features: The code is implemented on GPGPUs with significantly improved speed. Published by Elsevier B.V.

A reduced transport model for microtearing modes is developed for use in integrated predictive modeling studies, employing a unified fluid/kinetic approach to derive the nonlinear dispersion relation. This approach advances the kinetic description and allows the inclusion of nonlinear effects due to magnetic fluctuations. In this numerical study, the dependence of the microtearing mode real frequency and growth rate on plasma parameters and on DIII-D like L-mode and H-mode plasma profiles is examined. The magnetic fluctuation strength as well as electron thermal diffusivity due to microtearing modes is computed. The saturated amplitude of the magnetic fluctuations is calculated utilizing numerically determined microtearing mode eigenvalues in the nonlinear microtearing modes envelope equation. It is found that the electron temperature gradient in the presence of moderate collision frequency is required for the microtearing mode to become unstable. The effects of small and large collisionality and small and large wavenumbers on microtearing modes are found to be stabilizing, while the effects of density gradient, plasma beta, low current density, and large magnetic shear are found to be destabilizing. The microtearing mode growth rate, magnetic fluctuation strength, as well as electron thermal diffusivity is found to be larger in the H-mode plasma than in the L-mode plasma.

Microtearing modes (MTMs) have been identified as a source of significant electron thermal transport in tokamak discharges. In order to describe the evolution of these discharges, it is necessary to improve the prediction of electron thermal transport. This can be accomplished by utilizing a model for transport driven by MTMs in whole device predictive modeling codes. The objective of this paper is to develop the dispersion relation that governs the MTM driven transport. A unified fluid/kinetic approach is used in the development of a nonlinear dispersion relation for MTMs. The derivation includes the effects of electrostatic and magnetic fluctuations, arbitrary electron-ion collisionality, electron temperature and density gradients, magnetic curvature, and the effects associated with the parallel propagation vector. An iterative nonlinear approach is used to calculate the distribution function employed in obtaining the nonlinear parallel current and the nonlinear dispersion relation. The third order nonlinear effects in magnetic fluctuations are included, and the influence of third order effects on a multi-wave system is considered. An envelope equation for the nonlinear microtearing modes in the collision dominant limit is introduced in order to obtain the saturation level. In the limit that the mode amplitude does not vary along the field line, slab geometry, and strong collisionality, the fluid dispersion relation for nonlinear microtearing modes is found to agree with the kinetic dispersion relation.

Yawed turbines are known to cause wake deflection, which has been considered in wake controlling concepts for the purpose of reducing wake loss. However, an isolated yawed turbine can only produce a small amount of wake shifting for downstream turbines, resulting in an overall loss of power output. For this reason, yawing as a wake controlling method is often found to be ineffective. Based on our Large Eddy simulation (LES) results, it is discovered that wake deflection can be intensified by multiple yawed turbines in a line. While each turbine is yawed by a relatively small angle, collectively they can deflect the wake significantly to reduce wake loss of other turbines, so that the overall power output can be increased. An analysis on skew angle is carried out based on existing skew model and the challenges of combined wake skew prediction is identified.

A fine-grained block incomplete LU (FGBILU) factorization for solving large-scale block-sparse linear systems resulting from coupled PDE systems with n equations has been recently developed for massively parallel heterogeneous architectures, such as generalpurpose graphics processing units (GPGPUs). A straightforward one-sweep wavefront ordering is combined with element-wise block submatrix operations, allowing FGBILU to achieve low-overhead concurrent computation at O ( n2N2 ) scale on a 3D PDE domain with a linear scale of N . Numerical experiments show that FGBILU is less efficient on smaller domains. Besides the inevitable performance penalty of a wavefront ordering, the index reconstruction by each concurrent computation thread causes considerable parallelism overhead. One way to reduce the overhead is to employ thread recycling along with CUDA inter-block synchronization. Dynamic parallelism is also attempted, although with no significant perforamnce benefit. The improved FGBILU is tested for a series of 3D PDE domains extracted from an incompressible Navier-Stokes solver called INCOMP3D. Results show that thread recycling can significantly reduce parallelism overhead and improve the performance of FGBILU on smaller domains.

Iterative methods based on block incomplete LU (BILU) factorization are considered highly effective for solving large-scale block-sparse linear systems resulting from coupled PDE systems with n equations. However, efforts on porting implicit PDE solvers to massively parallel shared-memory heterogeneous architectures, such as general-purpose graphics processing units (GPGPUs), have largely avoided BILU, leaving their enormous performance potential unfulfilled in many applications where the use of implicit schemes and BILU-type preconditioners/solvers is highly preferred. Indeed, strong inherent data dependency and high memory bandwidth demanded by block matrix operations render naive adoptions of existing sequential BILU algorithms extremely inefficient on GPGPUs. In this study, we present a fine-grained BILU (FGBILU) scheme which is particularly effective on GPGPUs. A straightforward one-sweep wavefront ordering is employed to resolve data dependency. Granularity is substantially refined as block matrix operations are carried out in a true element-wise approach. Particularly, the inversion of diagonal blocks, a well-known bottleneck, is accomplished by a parallel in-place Gauss–Jordan elimination. As a result, FGBILU is able to offer low-overhead concurrent computation at O(n2N2) scale on a 3D PDE domain with a linear scale of N. FGBILU has been implemented with both OpenACC and CUDA and tested as a block-sparse linear solver on a structured 3D grid. While FGBILU remains mathematically identical to sequential global BILU, numerical experiments confirm its exceptional performance on an Nvidia GPGPU.

General-purpose computing on graphics processing units (GPGPU) is a massive fine-grain parallel computation platform, which is is particularly attractive for CFD tasks due to its potential of one or two magnitudes of performance improvement with relatively low capital investment. Many successful attempts have been reported in recent years (see, for example [1, 2, 3, 4, 5, 6]). Although early attempts of utilizing GPGPU for CFD has been hampered by the heterogeneous nature of GPGPU hardware and complex programming tools, recent GPU technology has seen significant improvement on programming toolchain. The emergence of OpenACC [7], a directive-based programming model closely resembling OpenMP, further reduces the obstacles for efficient GPGPU programming. By allowing programmers to use a collection of compiler directives to specify loops and regions of their codes to be offloaded from a host CPU to GPGPU, this programming model offers a good balance between porting efforts and performance gain. As shown in [8, 6], our previous attempts of porting existing CFD codes to GPGPU using OpenACC and MVAPICH2 [9] have proven the advantages on portability and maintainability of this approach. Even with the assistance of convenient development tools many of the challenges in GPGPU can be resolved, including memory contingency and efficient data exchange between multiple GPU’s. A more fundamental performance restriction is encountered when porting implicit time integration schemes. Employing an implicit time marching instead of explicit time marching can often significantly improve simulation speed. This is particularly true when the physical time scale is relatively large, as is often the case in incompressible flows. From a more general point of view, governing equations of many complex systems are so stiff that an explicit method may never converge under any reasonable CFL number. One prime example is multiphase reactive flows, which, due to their incorporation of a large number of variables and equations for describing the complex thermodynamical and chemical processes, often result in highly stiff linear systems. Another example is highly streched computational grids due to unusual boundary shapes of the domain, which also result in stiff linear systems. Even formulated as an implicit problem, the resulting system matrix is often non-diagonally

A GPU accelerated p-multigrid discontinuous Galerkin (DG) method based on the OpenACC directives is presented for compressible flows on 3-D unstructured grids. The present design is aimed to utilize the power of high-performance GPU computing with very little intrusion and algorithm alteration to a well-developed CPU-based code. Due to the fact that the GPU memory is still far from abundant for high-order DG methods even on a top-rank model, a p-multigrid technique is therefore preferred for convergence acceleration rather than an implicit algorithm that requires huge memory for storing high-order Jacobian matrices. In this study, a multi-stage explicit time stepping scheme is used for advancing the higher-order approximation in time, with a first-order matrix-free implicit backward Euler scheme applied to accelerate the lower-order approximation. A variety of inviscid flow problems are computed on an NVIDIA Tesla K20c GPU to assess the performance of the developed GPU-accelerated code using a strong scaling test. The numerical results indicate that the p-multigrid discontinuous Galerkin method can be effectively accelerated on GPU in comparison with two eight-core AMD Opteron-6128 CPUs for its CPU-based counterpart.

OpenACC is a directive-based programing standard aim to provide a highly portable programming model for massively-parallel accelerators, such as General-purpose Computing on Graphics Processing Units (GPGPU), Accelerated Processing Unit (APU) and Many Integrated Core Architecture (MIC). The heterogeneous nature of these accelerators stresses a demand for careful planning of data movement and novel approaches of parallel algorithms not commonly involved in scientific computation. By following a similar concept of OpenMP, the directive-based approach of OpenACC hides many underlying implementation details, thus significantly reduces the programming complexity and increases code portability. However, many challenges remain, due to the relatively narrow interconnection bandwidth among GPUs and the very fine granularity of GPGPU architecture. The first is particularly restrictive when cross-node data exchange is involved in a cluster environment. Furthermore, GPGPU’s fine-grained parallelism is in conflict with certain types of inherently serial algorithms, posing further restrictions on performance. In our study, an implicit multi-block incompressible Navier-Stokes solver is ported for GPGPU using OpenACC and MVAPICH2. A performance analysis is carried out based on the profiling of this solver running in a InfiniBand cluster with nVidia GPUs, which helps to identify the potentials of directive-based GPU programming and directions for further improvement.

A multi-GPU accelerated, third-order, reconstructed discontinuous Galerkin method, namely RDG(P1P2), has been developed based on the OpenACC directives for compressible flows on 3D hybrid grids. The present scheme requires minimum intrusion and algorithm alteration to an existing CPU code, which renders an efficient design approach for upgrading a legacy CFD solver with the GPU-computing capability while maintaining its portability across multiple platforms. The grid partitioning is performed according to the number of GPUs, and loaded equally on each GPU. Communication between the GPUs is achieved via the host-based MPI. A face renumbering and grouping algorithm is used to eliminate memory contention due to vectorized computing over the face loops on each individual GPU. A series of inviscid and viscous flow problems have been presented for the verification and scaling test, demonstrating excellent scalability of the resulting GPU code. The numerical results indicate that this parallel RDG(P1P2) method is a cost-effective, high-order DG method for scalable computing on GPU clusters.

A GPU-accelerated discontinuous Galerkin (DG) method is presented for the solution of compressible flows on 3-D unstructured grids. The present work has employed two of the most attractive features in a new programming standard of parallel computing – OpenACC: 1) multi-platform/compiler support and 2) descriptive directive interface to upgrade a legacy CFD solver with the capacity of GPU computing, without significant extra cost in recoding, resulting in a highly portable and extensible GPU-accelerated code. In addition, a face renumbering/grouping scheme is proposed to overcome the “race condition” in facebased flux calculations that occurs on GPU vectorization. Performance of the developed double-precision solver is assessed for both simple and complex geometries. Speedup factors up to but not limited to 24× and 1.6× were achieved by comparing the measured computing time of the OpenACC program running on an NVIDIA Tesla K20c GPU to that of the equivalent MPI program running on one single core and full sixteen cores of an AMD Opteron-6128 CPU respectively, indicating a great potential to port more features of the underlying DG solver into the OpenACC framework.