The use of closures, a core language feature of functional programming languages, has become popular in the context of concurrent and distributed programming. Using closures in a concurrent or distributed setting increases safety hazards, however, due to captured variables. Previous work proposed spores, enhanced closures that increase safety by constraining their environment using types. This paper presents Spores3, a completely new library-based implementation of spores for Scala 3. It is shown how the new design is enabled by a unique combination of several new features in Scala 3. Moreover, Spores3 contributes a new, portable approach to serializing closures based on type classes. Its implementation supports the same serialization approach on both the JVM and the JavaScript backends of Scala.
Regular expressions can easily go wrong. Capturing groups, in particular, require meticulous care to avoid running into off-by-one errors and null pointer exceptions. In this chapter, we propose a new design for Scala's regular expressions which completely eliminates this class of errors. Our design makes extensive use of match types, Scala's new feature for type-level programming, to statically analyze regular expressions during type checking. We show that our approach has a minor impact on compilation times, which makes it suitable for practical use.
The explicit nulls feature was merged into Scala 3 compiler at the end of 2019, which makes Null no longer a subtype of all reference types. This is the first step to enforce null safety in Scala language. Since then, we are continuously improving the usability to help users migrating to explicit nulls more easily. The UncheckedNull (originally named JavaNull) was introduced in the original design to allow calling Java members unsafely (Nieto et al. 2020). Due to limited usage and difficulty of implementation, we decided to discard this notion and introduced a new language feature, called UnsafeNulls. By simply importing scala.language-.unsafeNulls, users can create an "unsafe" scope. Inside this scope, Null will have similar semantic as in Scala without explicit nulls, which allows selecting members on nullable variables and assigning nullable values to non-nullable variables without checking. This is useful when a large chunk of Scala code is mainly interacting with nullable values from Java library. The community projects are used to evaluate this new feature. We found UnsafeNulls can significantly reduce the work of initial migration. This gives users more flexibility to migrate their projects gradually. We also migrated the Scala 3 compiler itself to explicit nulls.
Parser combinators are a popular and elegant approach for parsing in functional languages. The design and implementation of such libraries are well discussed, but having a well-designed library is only one-half of the story. In this paper we explore several reusable approaches to writing parsers in combinator style, focusing on easy to apply patterns to keep parsing code simple, separated, and maintainable.
High-performance computing research is entering the exascale computing era. Large-scale simulations have more than enough parallel resources but reach a saturation point in spatial parallelization due to the communication cost and synchronization overhead. Parallel-in-time (PinT) methods are a solution to the saturation problem. To date, many efficient PinT methods have been proposed, but most experiments focus on small problems with a large number of time steps. Applications that are solved by explicit time-marching schemes, especially, are highly scalable in the spatial domain. Therefore very few researchers use PinT methods on explicit time-marching schemes. Furthermore, PinT methods often require many processors to achieve faster computation than sequential code. This work proposes a PinT method for explicit schemes that provide efficient parallelization with few working processes. The numerical experiment of advection equation and compressible viscous flow simulation shows that the proposed method could achieve better parallel efficiency than spatial parallelization.
A new mixed-precision preconditioner based on the iterative refinement (IR) method is developed for the preconditioned conjugate gradient (P-CG) solver and the multigrid preconditioned conjugate gradient (MGCG) solver in the multiphase thermal-hydraulic CFD code JUPITER [1]. In the IR preconditioner, linear systems are approximately solved using a block Jacobi method, which is optimized using fine block decomposition with 2D tiles to facilitate SIMD operations and software pipelining. Mixed-precision computing is designed to avoid influences of the roundoff errors in computing ill-conditioned matrices with extreme contrast of coefficients. Linear systems are normalized so that it can be computed within the dynamic range of FP16. All data is stored in FP16 to reduce memory access, while all computation is performed in FP32. The hybrid FP16/32 implementation keeps the similar convergence property as FP32, while the computational performance is close to FP16. The robustness of the FP16/32 implementation was demonstrated by scanning data formats with different bit lengths of the significand. The developed solvers are optimized on Fugaku (A64FX), and applied to ill-conditioned matrices with 90 billion DOFs in JUPITER. The P-CG and MGCG solvers with the new IR preconditioner show excellent strong scaling up to 8,000 CPUs, where 5.7 × and 3.4 × speedups are respectively achieved from the conventional solvers on Oakforest-PACS (KNL).
In this paper, we present the results of our exploration of the applicability of preconditioners computed using the Markov Chain Monte Carlo Matrix Inversion ((MC)2MI) method to a variety of linear systems from the domain of quantum chromodynamics, plasma physics and engineering. The latter two are represented by matrices extracted from BOUT++ and Nektar++ codes with specific problem statements. Additionally, we present the scaling behaviour of our implementation for GPUs and CPUs.
This paper presents batched iterative solvers for GPU architectures. We elaborate on the design of the batched functionality aiming for optimal performance while still giving the user some flexibility in terms of choosing a sparse matrix format, a preconditioner optimized for the distinct items of the batch, and an application-specific stopping criterion that is evaluated for each problem in the batch, individually. Performance results for benchmark problems coming from PeleLM simulations reveal the potential of the batched iterative solvers for computational chemistry simulations, and their advantage compared to the current vendor-provided batched solutions.
The evolution of data science and machine learning has increased the applicability of the sparse matrix multiplication (SPGEMM) kernel. Unlike more well-known operations such as the SPMV, in the SPGEMM the nonzero pattern of the result is determined by the interaction between the nonzero patterns of the inputs, which impose serious challenges to the development of high-performance implementations for accelerators. Recent efforts in this subject aim to mitigate this irregularity through the use of block-based sparse storage formats, obtaining promissing results on accelerators such as GPUs. In this work we study the format bmSparse [1] and propose optimizations to attack the principal bottlenecks of the original SPGEMM implementation for Nvidia GPUs. We evaluate the proposal using nine sparse matrices of different sizes, showing remarkable speedups with respect to CUSPARSE's CSR variant.
Stochastic Gradient Descent (SGD) is a valuable algorithm for large-scale machine learning, but has proven difficult to parallelize on conventional architectures because of communication and memory access issues. The HogWild series of mixed logically distributed and physically multi-threaded algorithms overcomes these issues for problems with sparse characteristics by using multiple local model vectors with asynchronous atomic updates. While this approach has proven effective for several reported examples, there are others, especially very sparse cases, that do not scale as well. This paper discusses an SGD Support Vector Machine (SVM) on a cacheless migrating thread architecture using the Hogwild algorithms as a framework. Our implementations on this novel architecture achieved superior hardware efficiency and scalability over that of a conventional cluster using MPI. Furthermore these improvements were gained using naive data partitioning techniques and hardware with substantially less compute capability than that present in conventional systems.
Generalized Algebraic Data Types (GADT) are a popular programming language feature allowing advanced type-level properties to be encoded in the data types of a program. While Scala does not have direct support for them, GADT definitions can be encoded through Scala class hierarchies. Moreover, the Scala 3 compiler recently augmented its pattern matching capabilities to reason about such class hierarchies, making GADT-based programming practical in Scala. However, the current implementation can only reason about type parameters, but Scala’s type system also features singleton types and abstract type members (collectively known as path-dependent types), about which GADT-style reasoning is also useful and important. In this paper, we show how we extended the existing constraint-based GADT reasoning of the Scala 3 compiler to also consider path-dependent types, making Scala’s support for GADT programming more complete and bringing Scala closer to its formal foundations.
We describe a scheme for reflecting exceptions as capabilities in the Scala type system that keeps notational overhead to a minimum and avoids well-known problems with Java's checked exceptions framework. The scheme makes exceptions safer but not fully safe since the capability for throwing an exception may still yet escape its enclosing try block. To address this limitation, we also propose a type system which prevents capabilities from escaping.
Recent work on the DOT calculus successfully put core aspects of Scala on a sound foundation, but subtyping in DOT is structural and therefore not easily amenable to studying the parts of Scala that are deeply tied to its nominal subtyping system. On the other hand, the Featherweight Java calculus has proven to be a great basis for studying many aspects of Java and Java-like languages. Continuing this tradition, we present Pathless Scala: an extension of Featherweight Generic Java that closely models multiple inheritance and intersection types as they exist in the Scala language today. We define the semantics of Pathless Scala by erasing it to a simpler calculus in a way that closely models how Scala is compiled to Java bytecode in practice. More than a one-off, we believe that this calculus could be extended to describe many more features of Scala, although reconciling it with DOT remains an open problem.
Objects under initialization are fragile: some of their fields are not yet initialized. Consequently, accessing those uninitialized fields directly or indirectly may result in program crashes or abnormal behaviors at runtime. A newly created object goes through several states during its initialization, beginning with all fields being empty until all of them are filled. However, ensuring initialization safety statically, without manual annotation of initialization states in the source code, is a challenge, due to aliasing, virtual method calls and typestate polymorphism. In this work, we introduce a novel analysis based on abstract interpreters to ensure initialization safety. Compared to the previous approaches, our analysis is simpler and easier to extend, and it does not require any user annotations. The analysis is inter-procedural, context-sensitive and flow-insensitive, yet it has good performance thanks to local reasoning and heap monotonicity.
We present new GPU implementations of the tensor contractions arising from basis-related computations for high-order finite element methods. We consider both tensor and non-tensor bases. In the case of tensor bases, we introduce new kernels based on a series of fused device-level matrix multiplications (GEMMs), specifically designed to utilize the fast memory of the GPU. For non-tensor bases, we develop a tuned framework for choosing standard batch-BLAS GEMMs that will maximize performance across groups of elements. The implementations are included in a backend of the libCEED library. We present benchmark results for the diffusion and mass operators using libCEED integration through the MFEM finite element library and compare to those of the previously best-performing GPU backends for stand-alone basis computations. In tensor cases, we see improvements of approximately 10-30% for some cases, particularly for higher basis orders. For the non-tensor tests, the new batch-GEMMs implementation is twice as fast as what was previously available for basis function order greater than five and greater than approximately 105 degrees of freedom in the mesh; up to ten times speedup is seen for eighth-order basis functions.
In this work we look at employing communication avoiding techniques commonly used in Krylov methods in the context of exponential integrators for the solution of stiff partial differential equations. We choose an exponential integrator method based on polynomial approximations, as compared to those based on Krylov methods, to improve the possible strong scaling by reducing the possible all-reduce communications prevalent in iterative Krylov solvers. We implement this within the published TeaLeaf mini-app which is parallelised with MPI+OpenMP, and has an MPI+CUDA implementation. We assess the scalability of our implementations on AWE's Damson Bull Sequana X1000 system up to 1024 nodes (36,864 cores), AWE's Bullace system which has nodes with attached Nvidia V100 GPUs, and on the EPCC's Fulhame HPE Apollo 70. We find that our port of TeaLeaf using an exponential Euler method (CPEXI) scales well, in particular when configured in hybrid (MPI+OpenMP) where it achieves a parallel efficiency of 0.57 on 36,864 cores. This is better than a communication avoiding polynomially preconditioned Conjugate Gradient method (CP-PCG) which achieves 0.36 at the same core count. Our GPU experiments show an encouraging speedup and scaling, with minimal overhead.
Given the trend of supercomputers accumulating much of their compute power in GPU accelerators composed of thousands of cores and operating in streaming mode, global synchronization points become a bottleneck, severely confining the performance of applications. In consequence, asynchronous methods breaking up the bulk-synchronous programming model are becoming increasingly attractive. In this paper, we study a GPU-focused asynchronous version of the Restricted Additive Schwarz (RAS) method that employs preconditioned Krylov subspace methods as subdomain solvers. We analyze the method for various parameters such as local solver tolerance and iteration counts. Leveraging the multi-GPU architecture on Summit, we show that these two-stage methods are more memory and time efficient than asynchronous RAS using direct solvers. We also demonstrate the superiority over synchronous counterparts, and present results using one-sided CUDA-aware MPI on up to 36 NVIDIA V100 GPUs.
Encouraged by the requirement of high speed matrix computations and training deep neural networks, TensorCore was introduced in NVIDIA GPU to further accelerate matrix-matrix multiplication. It supports very fast half precision general matrix matrix multiplications (GEMMs), which is around 8x faster than single precision CUDA core GEMMs. So far the use of TensorCore GPU for matrix operations other than matrix-matrix multiplications is under developed. In this paper, we propose some efficient BLAS3 operations that exploits TensorCore. The experimental results show that the proposed algorithms outperform cublas corresponding routines and the naive TensorCore implementation with up to 4.7x speedup.
Gaussian elimination is a key technique for solving dense, non-symmetric systems of linear equations. Pivoting is used to ensure numerical stability but can introduce significant overheads. We propose replacing pivoting with recursive butterfly transforms (RBTs) and iterative refinement. RBTs use an FFT-like structure and randomized elements to provide an efficient, two-sided preconditioner for factoring. This approach was implemented and tested using Software for Linear Algebra Targeting Exascale (SLATE). In numerical experiments, our implementation was more robust than Gaussian elimination with no pivoting (GENP) but failed to solve all the problems solvable with Gaussian elimination with partial pivoting (GEPP). Furthermore, the proposed solver was able to outperform GEPP when distributed on GPU-accelerated nodes.
Based on the Green's functions that reflect mathematical properties of partial differential equations (PDE), we developed a novel preconditioner using neural networks (NNs) with high accuracy and small computational cost for improving the convergence property of an iterative implicit solver. As the dense and uniform computation involved in NNs are more efficient than that of the conventional PDE solver schemes, we could solve the time evolution of a highly heterogeneous problem in 4.26-fold shorter time compared to a typical PDE solver. The method is also suitable for use with low-precision arithmetic in NNs as the accuracy of the final solution is guaranteed. The localized property of NNs enable high scalability for solving large problems: a 5.57 trillion degrees-of-freedom problem was solved with 98.1% weak scalability up to 55,296 Fugaku nodes with 30.3 PFLOPS (16.2% of FP64 peak performance). This method can be used in various PDE-based simulations and has potential to make broad ripple effects in various fields.
