A new parallel variant of the multi-projection method designed to solve general sparse linear systems on hybrid distributed-memory computing systems with multicore nodes is presented. The multi-projection method is an additive domain decomposition method based on semi-aggregation techniques. In the proposed variant, the multi-projection method is used as a preconditioner to a parallel preconditioned pipelined variation of the BiCGSTAB iterative method. As a domain decomposition method, it requires partitioning the graph corresponding to the coefficient matrix of the linear system, where the quality of the partitioning significantly affects convergence behavior. Thus, two new graph partitioning schemes are presented and their performance is compared against METIS with respect to the multi-projection method. The proposed graph partitioning schemes are based on Mongoose, utilizing both an iterative approach for partitioning and a hybrid multilevel approach based on METIS. Numerical results regarding the convergence behavior, the parallel performance and the scalability of the proposed scheme for several problems are given.
Halo exchange is the task of communicating the boundaries of a subdivided computational grid with neighboring processes in distributed computations. With graphics processors augmenting the world's fastest supercomputers, data movement across heterogeneous memory hierarchies must be carefully orchestrated to leverage these accelerators at scale. In this work, we implement and optimize packing, inter-process communication, domain decomposition, and topology-aware process assignment integral to efficient halo exchange on heterogeneous systems using GPU-aware MPI. Our test cases include benchmarks of the halo exchange on one- to five-dimensional grids on both AMD and Nvidia graphics processors. We find that in multidimensional halo exchange, custom packing kernels provide up to two orders of magnitude speedup compared to MPICH and OpenMPI implementations based on MPI datatypes. Furthermore, hierarchical topology-aware process assignment provides more consistent performance across varying grid sizes on heterogeneous systems compared to simpler methods. We apply our methods improve the strong scaling of magnetohydrodynamics simulations, demonstrating 1.58 & times; speedup on high process counts in parallel computation and communication. Our results highlight the importance of kernel fusion for alleviating small kernel overheads on graphics processors even if it prohibits pipelining packing with communication.
The Sunway TaihuLight supercomputer has enabled significant advances across diverse scientific and engineering domains, particularly through large-scale numerical and data-driven applications. However, on its SW26010 processor, the absence of an efficient data-sharing mechanism among computing cores often limits the performance of these applications. This paper presents swAllies, a high-efficiency data-sharing library designed to enhance application performance on the SW26010 processor. swAllies builds upon the SW26010's register-level communication capabilities and integrates two strategies: a bandwidth-oriented strategy and a redundancy-oriented strategy. Both strategies enhance memory access efficiency by reducing redundant data movement and improving bandwidth utilization. Additional optimizations are incorporated to improve the library's performance and flexibility, while manual and genetic-algorithm-based parameter tuning further improves adaptability across workloads. On the 2D FD4 benchmark, swAllies, together with standard optimization techniques, achieves a 13.54 & times; speedup, reaching 85% of the theoretical upper bound. Across eight representative workloads, swAllies delivers up to 2.71 & times; speedup, demonstrating its effectiveness for real-world scientific applications on the Sunway TaihuLight.
Operational models of nuclear cloud rise are required to have short simulation times to facilitate timely response by decision makers and emergency personnel. Because of this, current models are low dimensional approximations and rely on parameterized variables tuned to historical U.S. nuclear test results. These simplifications limit the capacity of current models to account for complex interactions with the local environment. As an alternative, this study examines the use of a new multiscale atmospheric solver, the Energy Research and Forecasting model (ERF), for use in fallout modeling applications. By simulating the fully three dimensional and time varying atmosphere dynamics, including turbulence with large eddy simulation (LES), significantly fewer modeling assumptions are required. In this study, four historical U.S. nuclear tests with varying heights of burst were simulated in ERF using an idealized setup and the results agree favorably with both historical test data and previous simulation data from the Weather Research and Forecasting model (WRF). Furthermore, it was demonstrated that cloud rise simulations could be performed faster than real time by leveraging GPU accelerated high performance computing resources with a combination of adaptive time stepping and adaptive mesh refinement. This represents a significant improvement in capability for nuclear cloud rise simulations and may enable the use of ERF as an operational tool in emergency situations.
In this paper, we propose a randomized point-block Schwarz preconditioner for a heterogeneous computing system consisting of both CPUs and GPUs. The preconditioner can be used to accelerate a Newton-Krylov method for solving nonlinear algebraic system of equations arising from the discretization of multi-component partial differential equations. Roughly speaking, GPUs consist of a large number of light-weighted processing cores for whom the synchronization cost is high. On such a processor, the traditional algorithms for the subdomain LU factorization and the triangular solves in the Schwarz method are no longer efficient since they require a large number of synchronizations for some or all of the processing cores. The key steps of the proposed method involve (1) a randomized iterative method for the LU factorization of a matrix; (2) a randomized iterative triangular solver for the upper and lower triangular systems of equations; (3) a symbolic inverse of point-block diagonal matrices whose size is determined by the Warp-size of the GPU. In the realization of these steps, we employ a warp-based thread partitioning strategy with an adjustable factor that controls the degree of parallelism for the randomized computation between threads. The target problems are matrices with block structures and the block size is small enough so that the inverses of diagonal blocks are obtained symbolically. Two representative applications covering the discretized incompressible Navier-Stokes and hyperelasticity equations verify the preconditioner's performance in a Newton-Krylov solver, with a 12x + speedup on 8 GPUs over 8 CPU cores. The randomized method in the first case achieves 2x-5x gains over the deterministic cuSPARSE baseline and performs nearly comparable to the baseline in the second case. Cerebral artery blood flow simulation proves its superior scalability to the deterministic GPU implementation, attaining up to 45% parallel efficiency on 64 GPUs.
