DLA-Future implements an efficient GPU-enabled distributed eigenvalue solver using a software architecture based on the C++ std::execution concurrency proposal. The state-of-the-art linear algebra implementations LAPACK and ScaLAPACK were designed for legacy systems and employ fork-join parallelism, which can perform inefficiently on modern architectures. The benefits of task-based linear algebra implementations are significant. The reduction of synchronization points and the ease of overlapping computation with communication are two of the main benefits that lead to improved performance. In specific cases, the ability to schedule multiple algorithms concurrently yields a noticeable reduction of time-to-solution. We present the implementation of DLA-Future and the results on different types of systems starting from Piz Daint multicore and GPU partitions, moving to more recent architectures available in ALPS. The benchmark results are divided into two categories. The first contains a comparison of DLA-Future against widely used eigensolver implementations. The second category showcases the performance of the eigensolver in real applications. We present results generated with CP2K, where DLA-Future support was easily added thanks to the provided C API, which is compatible with the ScaLAPACK interface.

Improving the precision in particle physics predictions obtained from lattice simulations of quantum chromodynamics (QCD) requires extension of the interactions considered thus far, leading to additional computational demands. Most commonly used publicly available program packages for efficient simulations of Wilson discretization of the Dirac operator are highly scalable on CPU hardware. In order to be able to run efficiently on existing and upcoming hybrid architectures, one needs to rethink the current strategy for data types used at different stages of the simulation, most notably in frequent solves of the Dirac equation. We perform the first steps towards porting on GPUs of the three type of solvers used in the simulations of clover improved Wilson fermions: Conjugate Gradient, Schwarz preconditioned GCR solver, and a variant of the deflated solver. The analysis of the reduced precision data types' impact on the convergence of each solver indicates several possibilities for overall performance improvement.

OpenQ$^\star$D code has been used by the RC$^\star$ collaboration for the generation of fully dynamical QCD+QED gauge configurations with C$^\star$ boundary conditions. In this talk, optimization of solvers provided with the openQ$^\star$D package relevant for porting the code on GPU-accelerated supercomputing platforms is discussed. We present the analysis of the current implementations of the GCR solver preconditioned with Schwarz alternating procedure for ill-conditioned Dirac-operators. With the goal of enabling support for GPUs from various vendors, a novel method of adaptive CPU/GPU-hybrid implementation is proposed.

We present the first open release of the DCA++ project, a high-performance research software framework to solve quantum many-body problems with cutting edge quantum cluster algorithms. DCA++ implements the dynamical cluster approximation (DCA) and its DCA$^+$ extension with a continuous self-energy. The algorithms capture nonlocal correlations in strongly correlated electron systems, thereby giving insight into high-$T_c$ superconductivity. The code's scalability allows efficient usage of systems at all scales, from workstations to leadership computers. With regard to the increasing heterogeneity of modern computing machines, DCA++ provides portable performance on conventional and emerging new architectures, such as hybrid CPU-GPU, sustaining multiple petaflops on ORNL's Titan and CSCS' Piz Daint supercomputers. Moreover, we show how sustainable and scalable development of the code base has been achieved by adopting standard techniques of the software industry. These include employing a distributed version control system, applying test-driven development and following continuous integration.

We propose COSMA: a parallel matrix-matrix multiplication algorithm that is near communication-optimal for all combinations of matrix dimensions, processor counts, and memory sizes. The key idea behind COSMA is to derive an optimal (up to a factor of 0.03% for 10MB of fast memory) sequential schedule and then parallelize it, preserving I/O optimality. To achieve this, we use the red-blue pebble game to precisely model MMM dependencies and derive a constructive and tight sequential and parallel I/O lower bound proofs. Compared to 2D or 3D algorithms, which fix processor decomposition upfront and then map it to the matrix dimensions, it reduces communication volume by up to √ times. COSMA outperforms the established ScaLAPACK, CARMA, and CTF algorithms in all scenarios up to 12.8x (2.2x on average), achieving up to 88% of Piz Daint's peak performance. Our work does not require any hand tuning and is maintained as an open source implementation.

Hybrid platforms - those combining multicore and many-core processors - have made extensive inroads into high performance computing, with 101 of the top 500 supercomputers being based on such technology. The addition of coprocessors ensures higher memory bandwidth and peak performance at the price of special programming or meticulous optimization. But how effective is hybrid multicore in increasing productivity of real applications? The answer depends to no small extent on the efficacy of the programming environment - compilers, de-buggers, profilers, as well as black-box components, such as platform-specific numerical libraries - to port applications correctly and with good performance to the underlying hardware.In this paper, we recount several use cases of production applications being porting to CSCS hybrid architectures, explain the challenges and strengths / weaknesses of various tools employed. Admittedly some of our conclusions are subjective, but our hope is that our experiences will lead to improvement of the PE's deficiencies, and will inform the HPC community of what worked well and what did not. This paper arises from the PARCO15 mini-symposium entitled "Is the programming environment ready for hybrid supercomputers?"

We present a scalable implementation of the Linearized Augmented Plane Wave method for distributed memory systems, which relies on an efficient distributed, block-cyclic setup of the Hamiltonian and overlap matrices and allows us to turn around highly accurate 1000+ atom all-electron quantum materials simulations on clusters with a few hundred nodes. The implementation runs efficiently on standard multi-core CPU nodes, as well as hybrid CPU-GPU nodes. The key for the latter is a novel algorithm to solve the generalized eigenvalue problem for dense, complex Hermitian matrices on distributed hybrid CPU-GPU systems. Performance tests for Li-intercalated CoO2 supercells containing 1501 atoms demonstrate that high-accuracy, transferable quantum simulations can now be used in throughput materials search problems. While our application can benefit and get scalable performance through CPU-only libraries like ScaLAPACK or ELPA2, our new hybrid solver enables the efficient use of GPUs and shows that a hybrid CPU-GPU architecture scales to a desired performance using substantially fewer cluster nodes, and notably, is considerably more energy efficient than the traditional multi-core CPU only systems for such complex applications.

The adoption of hybrid GPU-CPU nodes in traditional supercomputing platforms such as the Cray-XK6 opens acceleration opportunities for electronic structure calculations in materials science and chemistry applications, where medium-sized generalized eigenvalue problems must be solved many times. These eigenvalue problems are too small to effectively solve on distributed systems, but can benefit from the massive compute performance concentrated on a single node, hybrid GPU-CPU system. However, hybrid systems call for the development of new algorithms that efficiently exploit heterogeneity and massive parallelism of not just GPUs, but of multi/many-core CPUs as well. Addressing these demands, we developed a novel algorithm featuring innovative: Fine grained memory aware tasks, Hybrid execution/scheduling, and Increased computational intensity}. The resulting eigensolvers are state-of-the-art in HPC, significantly outperforming existing libraries. We describe the algorithm and analyze its performance impact on applications of interest when different fractions of eigenvectors are needed by the host electronic structure code.

We present a new quantum cluster algorithm to simulate models of high-Tc superconductors. This algorithm extends current methods with continuous lattice self-energies, thereby removing artificial long-range correlations. This cures the fermionic sign problem in the underlying quantum Monte Carlo solver for large clusters and realistic values of the Coulomb interaction in the entire temperature range of interest. We find that the new algorithm improves time-to-solution by nine orders of magnitude compared to current, state of the art quantum cluster simulations. An efficient implementation is given, which ports to multi-core as well as hybrid CPU-GPU systems. Running on 18,600 nodes on ORNL's Titan supercomputer enables us to compute a converged value of Tc/t = 0.053±0.0014 for a 28 site cluster in the 2D Hubbard model with U/t = 7 at 10% hole doping. Typical simulations on Titan sustain between 9.2 and 15.4 petaflops (double precision measured over full run), depending on configuration and parameters used.

SUMMARYFor software to fully exploit the computing power of emerging heterogeneous computers, not only must the required computational kernels be optimized for the specific hardware architectures but also an effective scheduling scheme is needed to utilize the available heterogeneous computational units and to hide the communication between them. As a case study, we develop a static scheduling scheme for the tridiagonalization of a symmetric dense matrix on multicore CPUs with multiple graphics processing units (GPUs) on a single compute node. We then parallelize and optimize the Basic Linear Algebra Subroutines (BLAS)‐2 symmetric matrix‐vector multiplication, and the BLAS‐3 low rank symmetric matrix updates on the GPUs. We demonstrate the good scalability of these multi‐GPU BLAS kernels and the effectiveness of our scheduling scheme on twelve Intel Xeon processors and three NVIDIA GPUs. We then integrate our hybrid CPU‐GPU kernel into computational kernels at higher‐levels of software stacks, that is, a shared‐memory dense eigensolver and a distributed‐memory sparse eigensolver. Our experimental results show that our kernels greatly improve the performance of these higher‐level kernels, not only reducing the solution time but also enabling the solution of larger‐scale problems. Because such symmetric eigenvalue problems arise in many scientific and engineering simulations, our kernels could potentially lead to new scientific discoveries. Furthermore, these dense linear algebra algorithms present algorithmic characteristics that can be found in other algorithms. Hence, they are not only important computational kernels on their own but also useful testbeds to study the performance of the emerging computers and the effects of the various optimization techniques. Copyright © 2013 John Wiley & Sons, Ltd.

Today's high computational demands from engineering fields and complex hardware development make it necessary to develop and optimize new algorithms toward achieving high performance and good scalability on the next generation of computers. The enormous gap between the high-performance capabilities of GPUs and the slow interconnect between them has made the development of numerical software that is scalable across multiple GPUs extremely challenging. We describe and analyze a successful methodology to address the challenges—starting from our algorithm design, kernel optimization and tuning, to our programming model—in the development of a scalable high-performance generalized eigenvalue solver in the context of electronic structure calculations in materials science applications. We developed a set of leading edge dense linear algebra algorithms, as part of a generalized eigensolver, featuring fine grained memory aware kernels, a task based approach and hybrid execution/scheduling. The goal of the new design is to increase the computational intensity of the major compute kernels and to reduce synchronization and data transfers between GPUs. We report the performance impact on the generalized eigensolver when different fractions of eigenvectors are needed. The algorithm described provides an enormous performance boost compared to current GPU-based solutions, and performance comparable to state-of-the-art distributed solutions, using a single node with multiple GPUs.

The adoption of hybrid GPU-CPU nodes in traditional supercomputing platforms opens acceleration opportunities for electronic structure calculations in materials science and chemistry applications, where medium sized Hermitian generalized eigenvalue problems must be solved many times. The small size of the problems limits the scalability on a distributed memory system, hence they can benefit from the massive computational performance concentrated on a single node, hybrid GPU-CPU system. However, new algorithms that efficiently exploit heterogeneity and massive parallelism of not just GPUs, but of multi/many-core CPUs as well are required. Addressing these demands, we implemented a novel Hermitian general eigensolver algorithm. This algorithm is based on a standard eigenvalue solver, and existing algorithms can be used. The resulting eigensolvers are state-of-the-art in HPC, significantly outperforming existing libraries. We analyze their performance impact on applications of interest, when different fractions of eigenvectors are needed by the host electronic structure code.