The Portable Extensible Toolkit for Scientific Computation (PETSc) library provides scalable solvers for nonlinear time-dependent differential and algebraic equations and for numerical optimization via the Toolkit for Advanced Optimization (TAO). PETSc is used in dozens of scientific fields and is an important building block for many simulation codes. During the U.S. Department of Energy's Exascale Computing Project, the PETSc team has made substantial efforts to enable efficient utilization of the massive fine-grain parallelism present within exascale compute nodes and to enable performance portability across exascale architectures. We recap some of the challenges that designers of numerical libraries face in such an endeavor, and then discuss the many developments we have made, which include the addition of new GPU backends, features supporting efficient on-device matrix assembly, better support for asynchronicity and GPU kernel concurrency, and new communication infrastructure. We evaluate the performance of these developments on some pre-exascale systems as well as the early exascale systems Frontier and Aurora, using compute kernel, communication layer, solver, and mini-application benchmark studies, and then close with a few observations drawn from our experiences on the tension between portable performance and other goals of numerical libraries.

The motivation for an in-situ study of microstructural evolution of eutectic tin-bismuth alloy was to explain the observation that in the early stage of electromigration stressing, and under certain conditions, there is a detectable drop in solder joint electrical resistance before the resistance begins to rise. At lower solder temperatures and electric current densities, the period of decreasing resistance is longer. A tin-bismuth solder joint under high electric current stress develops three distinct microstructural regions: At the anode end, a bismuth layer accumulates; at the cathode end, bismuth particles get swept away leaving behind the tin-rich phase, while the middle section is expected to remain essentially the eutectic microstructure. In the early stage of electromigration stressing, before any substantial amount of bismuth has piled up at the anode and bismuth particles swept away from the cathode, two phenomena occur that potentially affect the joint resistance: One is the microstructure driving towards thermodynamic equilibrium predicted by the Sn-Bi phase diagram. The other is the coarsening of the microstructure with the enlargement of the bismuth particles dispersed in the tin-rich phase matrix. Both these phenomena can potentially decrease the joint electrical resistance. At a later stage, when the bismuth layer at the anode becomes continuous, the solder joint resistance begins to rise. The paper will describe an in-situ electromigration study of planar geometry, 30-μm thick eutectic tin-bismuth solder in a scanning electron microscope. The microstructural changes caused by the current stressing were studied to shed light on the physics of solder joint resistance changes during the early stage of solder joint electromigration. In particular, the contributions of the two phenomena, the microstructure driving to achieve thermodynamic equilibrium and the microstructure coarsening, to the lowering of the solder joint resistance, are reported and discussed.

This manual describes the use of the Portable, Extensible Toolkit for Scientific Computation (PETSc) and the Toolkit for Advanced Optimization (TAO) for the numerical solution of partial differential equations (PDEs) and related problems on high-performance computers.PETSc/TAO is a suite of data structures and routines that provide the building blocks for implementing large-scale application codes on parallel (and serial) computers.PETSc uses the MPI standard for all distributed memory communication.PETSc/TAO includes a large suite of parallel linear solvers, nonlinear solvers, time integrators, and optimizers that may be used in application codes written in Fortran, C, C++, and Python (via petsc4py; see Getting Started ).The library is organized hierarchically, enabling users to employ the abstraction level most appropriate for a particular problem.By using techniques of object-oriented programming, PETSc provides enormous flexibility for users.PETSc is a sophisticated set of software tools; it initially has a steeper learning curve than packages such as MATLAB or a simple subroutine library.In particular, for individuals without some experience programming in C, C++, Python, or Fortran and experience using a debugger such as gdb or lldb, it may require a significant amount of time to take full advantage of the features that enable efficient software use.However, the power of the PETSc design and the algorithms it incorporates makes the efficient implementation of many application codes simpler than "rolling them" yourself.• For many tasks, a package such as MATLAB is often the best tool; PETSc is not intended for the classes of problems for which effective MATLAB code can be written.• Several packages (listed on https://petsc.org/),built on PETSc, may satisfy your needs without requiring directly using PETSc.We recommend reviewing these packages' functionality before starting to code directly with PETSc.• PETSc can be used to provide a "MPI parallel linear solver" in an otherwise sequential or OpenMP parallel code.This approach can provide modest improvements in the application time by utilizing modest numbers of MPI processes.See PCMPI for details on how to utilize the PETSc MPI linear solver server.Since PETSc is under continued development, small changes in usage and calling sequences of routines will occur.PETSc has been supported for twenty-five years; see mailing list information on our website for information on contacting support.PETSc/TAO Users Manual, Release 3.21.0DM: Interfacing Between Solvers and Models/Discretizations details how a user's models and discretizations can easily be interfaced with the solvers by using the DM construct.Additional Information describes a variety of useful information, including profiling, the options database, viewers, error handling, and some details of PETSc design.Visual Studio Code, Eclipse, Emacs, and Vim users may find their development environment's options for searching in the source code are useful for exploring the PETSc source code.Details of this feature are provided in Developer Environments.Note to Fortran Programmers: In most of the manual, the examples and calling sequences are given for the C/C++ family of programming languages.However, Fortran programmers can use all of the functionality of PETSc from Fortran, with only minor differences in the user interface.PETSc for Fortran Users provides a discussion of the differences between using PETSc from Fortran and C, as well as several complete Fortran examples. Note to Python Programmers:To program with PETSc in Python, you need to enable Python bindings (i.e.petsc4py) with the configure option --with-petsc4py=1.See the PETSc installation guide for more details.$ mpiexec -n 8 ./petsc_program_namepetsc_optionsPETSc also provides a script that automatically uses the correct mpiexec for your configuration.$ $PETSC_DIR/lib/petsc/bin/petscmpiexec -n 8 ./petsc_program_namepetsc_options Certain options are supported by all PETSc programs.We list a few particularly useful ones below; a complete list can be obtained by running any PETSc program with the option -help.• -log_view -summarize the program's performance (see Profiling)• -fp_trap -stop on floating-point exceptions; for example divide by zero • -malloc_dump -enable memory tracing; dump list of unfreed memory at conclusion of the run, see Detecting Memory Allocation Problems and Memory Usage, MatCreate(MPI_Comm comm,Mat *A); VecCreate(MPI_Comm comm,Vec *x); KSPCreate(MPI_Comm comm,KSP *ksp); Chapter 2. The Solvers in PETSc/TAO PETSc/TAO Users Manual, Release 3.21.0 DMNETWORK -Creating vectors for networksSee Networks for discussion of creating vectors with DMNETWORK. Common vector functions and operationsOne can examine (print out) a vector with the command VecView(Vec x,PetscViewer v);To print the vector to the screen, one can use the viewer PETSC_VIEWER_STDOUT_WORLD, which ensures that parallel vectors are printed correctly to stdout.To display the vector in an X-window, one can use the default X-windows viewer PETSC_VIEWER_DRAW_WORLD, or one can create a viewer with the routine PetscViewerDrawOpen().A variety of viewers are discussed further in Viewers: Looking at PETSc Objects.To create a new vector of the same format and parallel layout as an existing vector, useVecDuplicate(Vec old,Vec *new);To create several new vectors of the same format as an existing vector, useVecDuplicateVecs(Vec old,PetscInt n,Vec **new);This routine creates an array of pointers to vectors.The two routines are useful because they allow one to write library code that does not depend on the particular format of the vectors being used.Instead, the subroutines can automatically create work vectors based on the specified existing vector.As discussed in Duplicating Multiple Vectors, the Fortran interface for VecDuplicateVecs() differs slightly.When a vector is no longer needed, it should be destroyed with the command VecDestroy(Vec *x);To destroy an array of vectors, use the command VecDestroyVecs(PetscInt n,Vec **vecs);Note that the Fortran interface for VecDestroyVecs() differs slightly, as described in Duplicating Multiple Vectors.It is also possible to create vectors that use an array the user provides rather than having PETSc internally allocate the array space.Such vectors can be created with the routines such as

Eutectic tin-bismuth solder has been identified in an iNEMI (International Electronics Manufacturing Initiative) roadmap as the solder of choice for 2nd level electronic packaging because of its desirable property of low melting temperature causing less warpage of the assemblies being soldered. Bismuth in Sn-Bi solder however has a high propensity to electromigrate leading to its segregation at the anode end of the solder joint, increasing the electrical resistance and potentially the brittleness of the joint. This paper reports on the electromigration testing of 60-µm high eutectic Sn-Bi bottom-terminated-component (BTC) solder joints on OSP (organic surface preservative) and ENIG (immersion gold on electroless nickel) surface finishes in 60, 80, and 100 °C ambients subjected to current densities in the 1-3 kA/cm 2 range. Electromigration rates were determined by measuring the rates of increase in solder joint electrical resistances. The results were captured on Arrhenius plots that can be used to predict electromigration lifetimes under any application condition. The rates of electromigration were found to be similar for BTC solder joints on OSP and ENIG surface finishes.

This paper discusses the key challenges and future research directions for privacy-preserving federated learning (PPFL), with a focus on its application to large-scale scientific artificial intelligence models, in particular, foundation models (FMs). PPFL enables collaborative model training across distributed datasets while preserving privacy—an important collaborative approach for science. We discuss the need for efficient and scalable algorithms to address the increasing complexity of FMs, particularly when dealing with heterogeneous clients. In addition, we underscore the need for developing advance privacy-preserving techniques, such as differential privacy, to balance privacy and utility in large FMs emphasizing fairness and incentive mechanisms to ensure equitable participation among heterogeneous clients. Finally, we emphasize the need for a robust software stack supporting scalable and secure PPFL deployments across multiple high-performance computing facilities. We envision that PPFL would play a crucial role to advance scientific discovery and enable large-scale, privacy-aware collaborations across science domains.

The Exascale Computing Project (ECP) Software Technology and Co-Design teams addressed the growing complexities in high-performance computing (HPC) by developing scalable software libraries and tools that leverage exascale system capabilities. As we enter the exascale era, the need for reusable, optimized software solutions that can handle the unique challenges posed by these systems becomes increasingly important. The primary challenges the ECP teams faced were to create software libraries and tools that are performant on exascale architectures and portable and usable across diverse hardware platforms. Efforts addressed issues related to concurrent execution, memory management, and the integration of heterogeneous computing resources, such as GPUs from multiple vendors. The ECP’s strategy involved a structured development process encompassing the creation, optimization, and deployment of software in collaboration with industry, academia, and national laboratories. The project was organized into several technical areas: co-design of domain-specific suites with target applications, programming models and runtimes, development tools, mathematical libraries, data and visualization tools, and software ecosystem and delivery mechanisms. ECP has successfully developed a large portfolio of software libraries and tools that demonstrate significant improvements in performance and scalability on exascale systems. These products have been integrated into the Department of Energy’s computing facilities, supporting various scientific applications and ensuring robust performance across different hardware setups. ECP advancements in software development for exascale computing highlight the importance of a collaborative and adaptive approach to handling next-generation HPC systems complexities. The lessons learned emphasize the need for continuous engagement with end-users and vendors, and the importance of maintaining a balance between innovation and practical implementation. Future efforts will focus on ensuring scalability, keeping pace with rapid hardware advancements, and further enhancing the interoperability and usability of the software ecosystem. Subsequent articles in this special issue provide in-depth discussions and case studies into specific library and tool efforts.

This manual describes the use of the Portable, Extensible Toolkit for Scientific Computation (PETSc) and the Toolkit for Advanced Optimization (TAO) for the numerical solution of partial differential equations and related problems on high-performance computers. PETSc/TAO is a suite of data structures and routines that provide the building blocks for the implementation of large-scale application codes on parallel (and serial) computers. PETSc uses the MPI standard for all distributed memory communication.

the mechanisms needed within parallel application codes, such as parallel matrix and vector assembly routines. The library is organized hierarchically, enabling users to employ the level of abstraction that is most appropriate for a particular problem. By using techniques of object-oriented programming, PETSc provides enormous flexibility for users. PETSc is a sophisticated set of software tools; as such, for some users it initially has a much steeper learning curve than a simple subroutine library. In particular, for individuals without some computer science background, experience programming in C, C++, python, or Fortran and experience using a debugger such as gdb or dbx, it may require a significant amount of time to take full advantage of the features that enable efficient software use. However, the power of the PETSc design and the algorithms it incorporates may make the efficient implementation of many application codes simpler than "rolling them" yourself. • For many tasks a package such as MATLAB is often the best tool; PETSc is not intended for the classes of problems for which effective MATLAB code can be written. • There are several packages (listed on http://www.mcs.anl.gov/petsc), built on PETSc, that may satisfy your needs without requiring directly using PETSc. We recommend reviewing these packages functionality before using PETSc. • PETSc should not be used to attempt to provide a "parallel linear solver" in an otherwise sequential code. Certainly all parts of a previously sequential code need not be parallelized but the matrix generation portion must be parallelized to expect any kind of reasonable performance. Do not expect to generate your matrix sequentially and then "use PETSc" to solve the linear system in parallel. Since PETSc is under continued development, small changes in usage and calling sequences of routines will occur. PETSc is supported; see http://www.mcs.anl.gov/petsc/miscellaneous/mailinglists.html for information on contacting support. A list of publications and web sites that feature work involving PETSc may be found at http://www.mcs.anl.gov/petsc/publications. We welcome any reports of corrections for this document at petsc-maint@mcs.anl.gov.

The Portable Extensible Toolkit for Scientific Computation (PETSc) library provides scalable solvers for nonlinear time-dependent differential and algebraic equations and for numerical optimization; it is used in dozens of scientific fields, and has been an important building block for many computational geoscience applications. Starting from the terascale era in the 1990s and continuing into the present dawn of the exascale era, a major goal of PETSc development has been achieving the scalability required to fully utilize leadership-class supercomputers. We will describe some of the algorithmic developments made during the era in which achieving inter-node scalability was the primary challenge to enabling extreme-scale computation, and then survey the challenges posed by the current era in which harnessing the abundant fine-scale parallelism within compute nodes -- primarily in the form of GPU-based accelerators -- has assumed at least equal importance. We will discuss how the PETSc design for performance portability addresses these challenges while stressing flexibility and extensibility by separating the programming model used by application code from that used by the library. Additionally, we will discuss recent developments in PETSc's communication module, PetscSF, that enable flexibility and scalable performance across large GPU-based systems while overcoming some of the difficulties posed by working directly with the Message Passing Interface (MPI) on such systems. A particular goal of this talk will be to go beyond simply describing the work performed to prepare PETSc and simulation codes that rely on it to run on exascale-class systems, but to enumerate the challenges we encountered and to share the essential lessons learned that can help other developers to prepare and optimize their high-performance scientific computing codes for the exascale era.

A novel planar solder geometry is described that allowed real-time. non-destructive monitoring of the rate of bismuth segregation at the anode in eutectic Sn-Bi solder joints of lengths in the range 180-450 µm at various current densities and temperatures while tracking the extent of electromigration by electrical resistance means. The rate of bismuth segregation was found to be somewhat proportional to the solder joint length indicating a probable Blech back-stress effect. Initially, the solder joint electrical resistance decreased followed by an increase. The period of decreasing electrical resistance was much less at higher current densities. The electromigration activation energy was determined to be 0.92 eV.

PetscSF, the communication component of the Portable, Extensible Toolkit for Scientific Computation (PETSc), is designed to provide PETSc's communication infrastructure suitable for exascale computers that utilize GPUs and other accelerators. PetscSF provides a simple application programming interface (API) for managing common communication patterns in scientific computations by using a star-forest graph representation. PetscSF supports several implementations based on MPI and NVSHMEM, whose selection is based on the characteristics of the application or the target architecture. An efficient and portable model for network and intra-node communication is essential for implementing large-scale applications. The Message Passing Interface, which has been the de facto standard for distributed memory systems, has developed into a large complex API that does not yet provide high performance on the emerging heterogeneous CPU-GPU-based exascale systems. In this article, we discuss the design of PetscSF, how it can overcome some difficulties of working directly with MPI on GPUs, and we demonstrate its performance, scalability, and novel features.

Science applications preparing for the exascale era are increasingly exploring in situ computations comprising of simulation-analysis-reduction pipelines coupled in-memory. Efficient composition and execution of such complex pipelines for a target platform is a codesign process that evaluates the impact and tradeoffs of various application- and system-specific parameters. In this article, we describe a toolset for automating performance studies of composed HPC applications that perform online data reduction and analysis. We describe Cheetah, a new framework for composing parametric studies on coupled applications, and Savanna, a runtime engine for orchestrating and executing campaigns of codesign experiments. This toolset facilitates understanding the impact of various factors such as process placement, synchronicity of algorithms, and storage versus compute requirements for online analysis of large data. Ultimately, we aim to create a catalog of performance results that can help scientists understand tradeoffs when designing next-generation simulations that make use of online processing techniques. We illustrate the design of Cheetah and Savanna, and present application examples that use this framework to conduct codesign studies on small clusters as well as leadership class supercomputers.

The applications being developed within the U.S. Exascale Computing Project (ECP) to run on imminent Exascale computers will generate scientific results with unprecedented fidelity and record turn-around time. Many of these codes are based on particle-mesh methods and use advanced algorithms, especially dynamic load-balancing and mesh-refinement, to achieve high performance on Exascale machines. Yet, as such algorithms improve parallel application efficiency, they raise new challenges for I/O logic due to their irregular and dynamic data distributions. Thus, while the enormous data rates of Exascale simulations already challenge existing file system write strategies, the need for efficient read and processing of generated data introduces additional constraints on the data layout strategies that can be used when writing data to secondary storage. We review these I/O challenges and introduce two online data layout reorganization approaches for achieving good tradeoffs between read and write performance. We demonstrate the benefits of using these two approaches for the ECP particle-in-cell simulation WarpX, which serves as a motif for a large class of important Exascale applications. We show that by understanding application I/O patterns and carefully designing data layouts we can increase read performance by more than 80 percent.

Hard and soft infrastructure work in tandem to accomplish a particular enterprise. Soft infrastructure – the formal and informal culture, institutions, standards, practices, and procedures that support an enterprise – is often more important to human endeavors than hard infrastructure. For example, a highway system’s hard infrastructure (e.g., the roads) could not satisfy its enterprise – effective transportation — without the corresponding soft infrastructure of regulations, policing, driver habits, maintenance crews, and so forth. The relative cost and importance of the different aspects of infrastructure evolves, particularly during the development stages, from planning, to construction, to commissioning, operation, maintenance, and upgrading. The design, development, support, and dissemination of computer software is an archetypal example of soft infrastructure, regularly being the tail that wags the dog of computer hardware (prototypical hard infrastructure). Quantifying soft infrastructure is more difficult than hard infrastructure. Thus, it is often ignored or under-emphasized when analyzing or proposing changes to human-developed systems. In particular, government understanding and funding of scientific soft infrastructure lags well behind that of hard infrastructure (for example, experimental devices such as accelerators). In fact, the funding for soft infrastructure often only flows as a by-product of the funding for tangentially-related hard infrastructure. This paper explores the soft infrastructure of the Portable Extensible Toolkit for Scientific computing (PETSc), which we dub the PETSc community. To address the needs of next-generation science the PETSc community is simultaneously engaged in developing additional features and algorithms (construction and commissioning), supporting a vibrant set of users and developers (operation), and maintaining and upgrading existing software. The informal PETSc community agglomeration across these activities is the most valuable asset of PETSc. While we briefly discuss some of the technology applied to PETSc’s development and support, the dominant story is that of PETSc community building and strengthening, while considering the lessons we have learned that may apply to other software development projects.

We present the Feature Tracking Kit (FTK), a framework that simplifies, scales, and delivers various feature-tracking algorithms for scientific data. The key of FTK is our simplicial spacetime meshing scheme that generalizes both regular and unstructured spatial meshes to spacetime while tessellating spacetime mesh elements into simplices. The benefits of using simplicial spacetime meshes include (1) reducing ambiguity cases for feature extraction and tracking, (2) simplifying the handling of degeneracies using symbolic perturbations, and (3) enabling scalable and parallel processing. The use of simplicial spacetime meshing simplifies and improves the implementation of several feature-tracking algorithms for critical points, quantum vortices, and isosurfaces. As a software framework, FTK provides end users with VTK/ParaView filters, Python bindings, a command line interface, and programming interfaces for feature-tracking applications. We demonstrate use cases as well as scalability studies through both synthetic data and scientific applications including tokamak, fluid dynamics, and superconductivity simulations. We also conduct end-to-end performance studies on the Summit supercomputer. FTK is open sourced under the MIT license: https://github.com/hguo/ftk.

The Portable Extensible Toolkit for Scientific computation (PETSc) library delivers scalable solvers for nonlinear time-dependent differential and algebraic equations and for numerical optimization.The PETSc design for performance portability addresses fundamental GPU accelerator challenges and stresses flexibility and extensibility by separating the programming model used by the application from that used by the library, and it enables application developers to use their preferred programming model, such as Kokkos, RAJA, SYCL, HIP, CUDA, or OpenCL, on upcoming exascale systems. A blueprint for using GPUs from PETSc-based codes is provided, and case studies emphasize the flexibility and high performance achieved on current GPU-based systems.

A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth make it increasingly infeasible to store and share all data produced by scientific simulations. Instead, we need methods for reducing data volumes: ideally, methods that can scale data volumes adaptively so as to enable negotiation of performance and fidelity tradeoffs in different situations. Multigrid-based hierarchical data representations hold promise as a solution to this problem, allowing for flexible conversion between different fidelities so that, for example, data can be created at high fidelity and then transferred or stored at lower fidelity via logically simple and mathematically sound operations. However, the effective use of such representations has been hindered until now by the relatively high costs of creating, accessing, reducing, and otherwise operating on such representations. We describe here highly optimized data refactoring kernels for GPU accelerators that enable efficient creation and manipulation of data in multigrid-based hierarchical forms. We demonstrate that our optimized design can achieve up to 250 TB/s aggregated data refactoring throughput -- 83% of theoretical peak -- on 1024 nodes of the Summit supercomputer. We showcase our optimized design by applying it to a large-scale scientific visualization workflow and the MGARD lossy compression software.

An encoder-decoder neural network has been used to examine the possibility for acceleration of a partial integro-differential equation, the Fokker-Planck-Landau collision operator. This is part of the governing equation in the massively parallel particle-in-cell code, XGC, which is used to study turbulence in fusion energy devices. The neural network emphasizes physics-inspired learning, where it is taught to respect physical conservation constraints of the collision operator by including them in the training loss, along with the L2 loss. In particular, network architectures used for the computer vision task of semantic segmentation have been used for training. A penalization method is used to enforce the "soft" constraints of the system and integrate error in the conservation properties into the loss function. During training, quantities representing the density, momentum, and energy for all species of the system is calculated at each configuration vertex, mirroring the procedure in XGC. This simple training has produced a median relative loss, across configuration space, on the order of 10E-04, which is low enough if the error is of random nature, but not if it is of drift nature in timesteps. The run time for the Picard iterative solver of the operator scales as order n squared, where n is the number of plasma species. As the XGC1 code begins to attack problems including a larger number of species, the collision operator will become expensive computationally, making the neural network solver even more important, since the training only scales as n. A wide enough range of collisionality is considered in the training data to ensure the full domain of collision physics is captured. An advanced technique to decrease the losses further will be discussed, which will be subject of a subsequent report. Eventual work will include expansion of the network to include multiple plasma species.