Artificial intelligence (AI) has the potential for vast societal and economic gain; yet applications are developed in a largely ad hoc manner, lacking coherent, standardized, modular, and reusable infrastructures. The NSF-funded Intelligent CyberInfrastructure with Computational Learning in the Environment AI Institute ("ICICLE") aims to fundamentally advance edge-to-center, AI-as-a-Service, achieved through intelligent cyberinfrastructure (CI) that spans the edge-cloud-HPC computing continuum, plug-and-play next-generation AI and intelligent CI services, and a commitment to design for broad accessibility and widespread benefit. This design is foundational to the institute's commitment to democratizing AI. The institute's CI activities are informed by three high-impact domains: animal ecology, digital agriculture, and smart foodsheds. The institute's workforce development and broadening participation in computing efforts reinforce the institute's commitment to democratizing AI. ICICLE seeks to serve as the national nexus for AI and intelligent CI, and welcomes engagement across its wide set of programs.

Although direct GPU-to-GPU communication has been possible in MPI libraries for over a decade, the limited availability of compatible hardware at academic HPC centers has discouraged the development of algorithms in scientific applications that take advantage of this capability. In this paper, we take Amber, a molecular dynamics code used to simulate proteins and nucleic acids, as a test case. We demonstrate the modifications necessary to implement GPU-to-GPU communication. Compared to the previous implementation, these modifications show an average of approximately 36% improvement in performance overall and 84% for the important explicit solvent subset of the benchmarks.

This document describes a two-day meeting held for the Principal Investigators (PIs) of NSF CyberTraining grants. The report covers invited talks, panels, and six breakout sessions. The meeting involved over 80 PIs and NSF program managers (PMs). The lessons recorded in detail in the report are a wealth of information that could help current and future PIs, as well as NSF PMs, understand the future directions suggested by the PI community. The meeting was held simultaneously with that of the PIs of the NSF Cyberinfrastructure for Sustained Scientific Innovation (CSSI) program. This co-location led to two joint sessions: one with NSF speakers and the other on broader impact. Further, the joint poster and refreshment sessions benefited from the interactions between CSSI and CyberTraining PIs.

The lightning talks at EduHPC provide an opportunity to share early results and insights on parallel and distributed computing (PDC) education and training efforts. The four lightning talks at EduHPC 2022 cover a range of topics in broadening PDC education: (i) curriculum development efforts for the European Masters in HPC program, (ii) bootcamps for CI professionals who support the running of AI workloads on HPC systems, (iii) a GPU programming course following the Carpentries model and (iv) peer-review assignments to help students write efficient parallel algorithms within sustainable software libraries.

Soil infrared spectroscopy has great potential for estimating soil properties, but reference soil measurements are typically required in combination with multivariate statistical models to estimate soil properties. User-friendly predictive tools based on open-source statistical environment remain one of the main limitations to enable technology diffusion to non-specialist users. Our aim is to build capacity for an automated machine learning routine for rapid and robust prediction of soil health indicators using lab acquired soil infrared spectra. This intelligent system runs on R statistical environment and includes (1) a diverse soil spectral library comprising main physiographic regions from the USA Midwest region under diverse land uses and various sampling depths, (2) a classification process to detect potential outliers in newly acquired spectra using supervised machine learning techniques, and (3) a multi-model optimized prediction process based on linear and non-linear statistical procedures (partial least squares, support vector machines, and neural network). This prediction system works at the intersection of soil and data science and high-performance computing to enable efficient parallel processing of spectral data on multi-core coprocessors. Using artificial intelligence to automate soil infrared spectroscopy is a fundamental demand that will make this technique an effective routine in soil laboratories to estimate soil health.

Understanding the full-stack performance trade-offs and interplay among HPC applications, MPI libraries, the communication fabric, and the job scheduler is a challenging endeavor. Unfortunately, existing profiling tools are disjoint and only focus on profiling one or a few levels of the HPC stack limiting the insights they can provide. In this paper, we propose a standardized approach to facilitate near real-time, low overhead performance characterization, profiling, and evaluation of communication of high-performance communication middleware as well as scientific applications using a cross-stack approach by INAM. The profiling capabilities are supported in two modes of with and without modifications to the application depending on the scope of the profiling session. We design and implement our designs using an MPI_T-based standardized method to obtain near real-time insights for MPI applications at scales of up to 4,096 processes with less than 5% overhead. Through experimental evaluations of increasing batch size for DL training, we demonstrate novel benefits of INAM for cross-stack communication analysis in real-time to detect bottlenecks and resolve them, achieving up to 3.6x improvements for the use-case study. The proposed solutions have been publicly released with the latest version of INAM and currently being used in production at various HPC supercomputers.

The widespread application and success of computational and data intensive research approaches in every discipline represented on our campuses has resulted in a rapid proliferation of organizations, technologies, and professions affiliated in different ways with the support and advancement of activities related to research computing and data (RCD). While most agree that this growth is helping to advance numerous disciplines, the proliferation of organizations seeking to support, promote, and advance RCD has led to some challenges. Specifically, a lack of understanding and consensus concerning which organizations should be considered a part of RCD support hampers our ability to encourage collaborations among its complementary constituents, leads to unneeded and redundant activities, and makes it difficult to identify strategic priorities and address gaps where specific needs are not being met to advance various disciplinary activities. In this paper we introduce the ecosystem metaphor to help characterize the rapidly changing relationships among the growing set of organizations that in some way support and enable activities related to RCD. The ecosystem concept lends itself well to describing the many entities related to RCD because it emphasizes the larger system over its individual component parts and highlights their interdependence, while explicitly expecting their change over time. Our work to characterize the current RCD ecosystem, while imperfect, will serve as a foundation and framework for the development of a more complete view of the ever-changing RCD ecosystem. A more complete view of the RCD ecosystem will in turn help to advance the broad goals of its members by helping to foster and accelerate new and meaningful collaborations among them.

Designing a scalable real-time monitoring and profiling tool with low overhead for network analysis and introspection capable of capturing all relevant network events is a challenging task. Newer set of challenges come out as HPC systems are becoming larger and users are expecting to have better capabilities like real-time profiling at fine granularity. We take up this challenge by redesigning OSU INAM and making it capable to gather, store, retrieve, visualize, and analyze network metrics for large and complex HPC clusters. The enhanced OSU INAM tool provides scalability, low overhead and fined-granularity InfiniBand port counter inquiry and fabric discovery for HPC users, system administrators, and HPC developers. Our experiments show that, for a cluster of 1,428 nodes and 114 switches, the proposed design can gather fabric metrics at very fine (sub-second) granularity and discovers the complete network topology in approximately 5 minutes. The proposed design has been released publicly as a part of OSU INAM Tool and is available for free download and use from the project website.

Most current parallel applications use data parallelism with a combination of MPI and threads. There has been growing interest in task-based programming models, but little work has been done on hybrid MPI+task codes. In this paper, we describe our experiences implementing a hybrid MPI+Kokkos tasks programming model for two applications. The first application is a general matrix multiplication code and the second application is a breadth-first search.

Automated software-installation tools such as EasyBuild and Spack have enabled HPC support staff to more quickly deploy software environments for HPC. However, to provide reliable user-facing software environments, any change made to the environments should be recorded and tested. In this paper, we will describe a continuous integration-based framework for software management in use at the Ohio Supercomputer Center (OSC). In this framework, we rely on homegrown tools for automated software build and installation and the ReFrame framework to setup the testing system which can be triggered on commit to a local Gitlab repository. We will document the workflow and explain how we can deploy the software environment quickly and confidently using these tools.

The GooFit highly parallel fitting package for GPUs and CPUs has been substantially upgraded in the past year. Python bindings have been added to allow simple access to the fitting configuration, setup, and execution. A Python tool to write custom GooFit code given a (compact and elegant) MINT3/AmpGen amplitude description allows the corresponding C++ code to be written quickly and correctly. New PDFs have been added. The most recent release was built on top of the December 2017 2.0 release that added easier builds, new platforms, and a more robust and efficient underlying function evaluation engine.

Regression testing of HPC systems is of crucial importance when it comes to ensure the quality of service offered to the end users. At the same time, it poses a great challenge to the systems and application engineers to continuously maintain regression tests that cover as many aspects as possible of the user experience. In this paper, we briefly present ReFrame, a framework for writing regression tests for HPC systems and how this is used by CSCS, NERSC and OSC to continuously test their systems. ReFrame is designed to abstract away the complexity of the interactions with the system and to separate the logic of a regression test from the low-level details, which pertain to the system configuration and setup. Regression tests in ReFrame are simple Python classes that specify the basic parameters of the test plus any additional logic. The framework will load the test and send it down a well-defined pipeline which will take care of its execution. ReFrame can be easily set up on any cluster and its straightforward invocation allows it to be easily integrated with common continuous integration/deployment (CI/CD) tools, in order to perform continuous testing of an HPC system. Finally, its ability to feed the collected performance data to well known log channels, such as Syslog, Graylog or, simply, parsable log files, make it also a powerful tool for continuously monitoring the health of the system from user’s perspective.

Machine learning is an important applied research area in particle physics, beginning with applications to high-level physics analysis in the 1990s and 2000s, followed by an explosion of applications in particle and event identification and reconstruction in the 2010s. In this document we discuss promising future research and development areas in machine learning in particle physics with a roadmap for their implementation, software and hardware resource requirements, collaborative initiatives with the data science community, academia and industry, and training the particle physics community in data science. The main objective of the document is to connect and motivate these areas of research and development with the physics drivers of the High-Luminosity Large Hadron Collider and future neutrino experiments and identify the resource needs for their implementation. Additionally we identify areas where collaboration with external communities will be of great benefit.

Code improvements have been made to the Surface Extraction from TIN based Search-space Minimization (SETSM) software, used to produce digital elevation maps from satellite imagery. The improvements include increased code portability and stability, along with enhanced performance. The code has been made available on a public repository.

The GooFit package provides physicists a simple, familiar syntax for manipulating probability density functions and performing fits, and is highly optimized for data analysis on NVIDIA GPUs and multithreaded CPU backends. GooFit was updated to version 2.0, bringing a host of new features. A completely revamped and redesigned build system makes GooFit easier to install, develop with, and run on virtually any system. Unit testing, continuous integration, and advanced logging options are improving the stability and reliability of the system. Developing new PDFs now uses standard CUDA terminology and provides a lower barrier for new users. The system now has built-in support for multiple graphics cards or nodes using MPI, and is being tested on a wide range of different systems. GooFit also has significant improvements in performance on some GPU architectures due to optimized memory access. Support for time-dependent four-body amplitude analyses has also been added.

PGAS models with a lightweight synchronization and shared memory abstraction, are seen as a good alternative to the Message Passing model for irregular communication patterns. OpenSHMEM is a library based PGAS model. OpenSHMEM 1.3 introduced Non-Blocking data movement operations to provide better asynchronous progress and overlap. In this paper, we present our experiences in designing Non-Blocking Put and Get operations on InfiniBand systems. Using the MVAPICH2-X runtime, we present the alternative designs for intra-node and inter-node operations. We also present a set of new benchmarks to analyze the latency, message rate performance, and communication/computation overlap benefits. The performance evaluation shows 7X improvement in the message rate. Furthermore, using a 3D-Stencil based application kernel, we assess the benefits of OpenSHMEM Non-Blocking extensions. We show 50% and 28% improvement on 27 and 64 processes, respectively.

Data intensive collective operations have a notable impact on the execution time and consequently the energy consumption of HPC applications owing to the amount of memory/processor/network resources involved in the data movement. However, mechanisms such as offload and one-sided transfers that are backed by RDMA-enabled interconnects like InfiniBand along with modern transport protocols like Dynamic Connected (DC) provide new ways to express collective communication. Despite this, researchers have not considered the use of an RDMA-based algorithm backed by the appropriate transport protocol for reducing energy consumption of collective operations. In this paper, we take up this challenge and study the impact that RDMA and transport protocol aware designs can have on the processor/memory energy and performance of dense collective operations like All-to-all. Through evaluation, we also identify that while a single transport protocol may bring both performance and energy benefits for one application it may not do so consistently for all applications. Motivated by this, we propose designs that yield both benefits for all evaluated applications. Experimental evaluation shows that our proposed designs are able to deliver up to 1.7X savings in energy with little or no degradation in the communication performance for All-to-all collective operations on modern HPC systems.

Several techniques have been proposed in the past for designing non-blocking collective operations on high-performance clusters. While some of them required a dedicated process/thread or periodic probing to progress the collective others needed specialized hardware solutions. The former technique, while applicable to any generic HPC cluster, had the drawback of stealing CPU cycles away from the compute task. The latter gave near perfect overlap but increased the total cost of the HPC installation due to need for specialized hardware and also had other drawbacks that limited its applicability. On the other hand, the Remote Direct Memory Access technology and high performance networks have been pushing the envelope of HPC performance to multi-petaflop levels. However, no scholarly work exists that explores the impact such RDMA technology can bring to the design of non-blocking collective primitives. In this paper, we take up this challenge and propose efficient designs of personalized non-blocking collective operations on top of the basic RDMA primitives. Our experimental evaluation shows that our proposed designs are able to deliver near perfect overlap of computation and communication for personalized collective operations on modern HPC systems at scale. At the microbenchmark level, the proposed RDMA-Aware collectives deliver improvements in latency of up to 89 times for MPI_Igatherv, 3.71 times for MPI_Ialltoall and, 3.23 times for MPI_Iscatter over the state-of-the-art designs. We also observe an improvement of up to 19 % for the P3DFFT kernel at 8,192 cores on the Stampede supercomputing system at TACC.

The MPI programming model has been widely used for scientific applications. The emergence of Partitioned Global Address Space (PGAS) programming models presents an alternative approach to improve programmability. With the global data view and lightweight communication operations, PGAS has the potential to increase the performance of scientific applications at scale. However, since the PGAS models are emerging, it is unlikely that entire applications will be re-written with them. Instead, unified communication runtimes have paved the way for a new class of hybrid applications that can leverage the benefits of both MPI and PGAS models. In this paper, we re-design an existing MPI based scientific mini-application (MiniMD) with MPI and OpenSHMEM programming models. We propose two alternative designs using MPI and OpenSHMEM programming models and compare performance and scalability of those designs with the original MPI-based implementation. Our performance evaluations using MVAPICH2-X (Unified MPI+PGAS Communication Runtime over InfiniBand) show a 17% reduction in total execution time, compared to existing MPI-based design with 1,024 cores.