Abstract. Cloud microphysics is critical for weather and climate prediction. In this work, we document updates and corrections to the cloud microphysical scheme used in the Community Earth System Model (CESM) and other models. These updates include a new nomenclature for the scheme, now called Parameterization of Unified Microphysics Across Scales (PUMAS), and the ability to run the scheme on graphics processing units (GPUs). The main science changes include refactoring an ice number limiter and associated changes to ice nucleation, adding vapor deposition onto snow, and introducing an implicit numerical treatment for sedimentation. We also detail the improvements in computational performance that can be achieved with GPU acceleration. We then show the impact of these scheme changes on the (a) mean state climate, (b) cloud feedback response to warming, and (c) aerosol forcing. We find that corrections are needed to the immersion freezing parameterization and that ice nucleation has important impacts on climate. We also find that the revised scheme produces less cloud liquid and ice but that this can be adjusted by changing the loss process for cloud liquid (autoconversion). Furthermore, there are few discernible effects of the PUMAS changes on cloud feedbacks but some reductions in the magnitude of aerosol–cloud interactions (ACIs). Small cloud feedback changes appear to be related to the implicit sedimentation scheme, with a number of factors affecting ACIs.

Abstract. The complexity of each Coupled Model Intercomparison Project grows with every new generation. The Phase 5 effort saw a large increase in the number of experiments that were performed and the number of variables that were requested compared to its previous generation, Phase 3. Many centers were not prepared for the large demand and this stressed the resources of several centers including at the National Center for Atmospheric Research. During Phase 5, we missed several deadlines and we struggled to get the data out to the community for analysis. In preparation for the current generation, Phase 6, we examined the weaknesses in our workflow and addressed the performance issues with new software tools. Through this investment, we were able to publish approximately six times the amount of data to the community compared to the volumes we produced in the previous generation and we were able to accomplish this within one-third of the time, providing an 18 times speedup. This paper discusses the improvements we have made to accomplish this success for Phase 6 and further improvements we hope to make for the next generation.

The complexity of each Coupled Model Intercomparison Project grows with every new generation. The Phase 5 effort saw a dramatic increase in the number of experiments that were performed and the number of variables that were requested compared to its previous generation, Phase 3. The large increase in data volume stressed the resources of several centers including at the National Center for Atmospheric Research. During Phase 5, we missed several deadlines and we struggled to get the data out to the community for analysis. In preparation for the current generation, Phase 6, we examined the weaknesses in our workflow and addressed the performance issues with new software tools. Through this investment, we were able to publish approximately 565 TB of compressed data to the community, with another 30 TB yet to be published. When compared to the volumes we produced in the previous generation, 165 TB of uncompressed data, we were able to provide 6 times the amount of data and we accomplish this within one-third of the time. This provided us with an approximate 18 times faster speedup. While this paper discusses the improvements we have made to our own workflow for the Coupled Model Intercomparison Project Phase 6 (CMIP6), we hope to encourage other centers to evaluate and invest in their own workflows in order to be successful in these types of modeling campaigns.

AbstractWe present an unprecedented set of high‐resolution climate simulations, consisting of a 500‐year pre‐industrial control simulation and a 250‐year historical and future climate simulation from 1850 to 2100. A high‐resolution configuration of the Community Earth System Model version 1.3 (CESM1.3) is used for the simulations with a nominal horizontal resolution of 0.25° for the atmosphere and land models and 0.1° for the ocean and sea‐ice models. At these resolutions, the model permits tropical cyclones and ocean mesoscale eddies, allowing interactions between these synoptic and mesoscale phenomena with large‐scale circulations. An overview of the results from these simulations is provided with a focus on model drift, mean climate, internal modes of variability, representation of the historical and future climates, and extreme events. Comparisons are made to solutions from an identical set of simulations using the standard resolution (nominal 1°) CESM1.3 and to available observations for the historical period to address some key scientific questions concerning the impact and benefit of increasing model horizontal resolution in climate simulations. An emerging prominent feature of the high‐resolution pre‐industrial simulation is the intermittent occurrence of polynyas in the Weddell Sea and its interaction with an Interdecadal Pacific Oscillation. Overall, high‐resolution simulations show significant improvements in representing global mean temperature changes, seasonal cycle of sea‐surface temperature and mixed layer depth, extreme events and in relationships between extreme events and climate modes.

Two high‐resolution versions of a Coupled Earth System Model (CESM1.3: 0.25° atmosphere, 1° ocean; CESM1.1: 0.25° atmosphere, 0.1° ocean) are compared to the standard resolution CESM1.1 and CESM1.3 (1° atmosphere, 1° ocean). The CESM1.3 versions are documented, and the consequences of model resolution, air‐sea coupling, and physics in the atmospheric models are studied with regard to storm tracks in the Southern Hemisphere as represented by 850‐hPa eddy kinetic energy. Increasing the resolution from 1° to 0.25° in the atmosphere (same physics) coupled to the 1° ocean intensifies the strength of the storm tracks closer to observations. The 0.25° atmosphere with the older CESM1.1 physics coupled to the 0.1° ocean has fewer low clouds, warmer Southern Ocean sea surface temperatures, a weaker meridional temperature gradient, and a degraded storm track simulation compared to the 0.25° atmosphere with CESM1.3 physics coupled to the 1° ocean. Therefore, deficient physics in the atmospheric model can negate the gains attained by higher resolution in atmosphere and ocean.

It is the purpose of this paper to provide a comprehensive documentation of the new NCAR (National Center for Atmospheric Research) version of the spectral element (SE) dynamical core as part of the Community Earth System Model (CESM2.0) release. This version differs from previous releases of the SE dynamical core in several ways. Most notably the hybrid sigma vertical coordinate is based on dry air mass, the condensates are dynamically active in the thermodynamic and momentum equations (also referred to as condensate loading), and the continuous equations of motion conserve a more comprehensive total energy that includes condensates. Not related to the vertical coordinate change, the hyperviscosity operators and the vertical remapping algorithms have been modified. The code base has been significantly reduced, sped up, and cleaned up as part of integrating SE as a dynamical core in the CAM (Community Atmosphere Model) repository rather than importing the SE dynamical core from High-Order Methods Modeling environment as an external code.

A kernel or mini-app is a self-contained small application that retains certain characteristics of the original application [7]. Working on a kernel or mini-app in the place of the original application can dramatically reduce the resources and effort required for performing software tasks such as performance optimization and porting to new platforms. However, using kernel as a proxy is based on the assumption that it represents the original application in the context of how it is being used. In this paper, we introduce an extension to the Fortran Kernel Generator (KGen) which is an automated kernel extraction tool [1]. The extension allows comparison of the execution characteristics between the original application and the generated kernel using descriptive statistics. From the comparison, the user is provided with statistics that provide information on the degree and context of representativeness of the kernel. KGen also utilizes the information generated to help it to automatically improve representativeness of the kernels whilst reducing the size of the workload generated. We applied this extension to three kernels. One is generated from a Fortran scientific library and the remaining two are generated from an earth system model. We have demonstrated that the descriptive statistics provided in the enhancement provide not only quantitative metrics and context of representativeness but also a way to improve the quality of representativeness of the kernels generated.

This tutorial presents the details of the interconnection network utilized in many high performance computing (HPC) systems today. “InfiniBand” is the hardware interconnect utilized by over 35% of the top 500 supercomputers in the world as of June, 2017. “Verbs” is the term used for both the semantic description of the interface in the InfiniBand architecture specifications, and the name used for the functions defined in the widely used OpenFabrics alliance implementation of the software interface to InfiniBand. “Message passing interface” is the primary software library by which HPC applications portably pass messages between processes across a wide range of interconnects including InfiniBand. Our goal is to explain how these three components are designed and how they interact to provide a powerful, efficient interconnect for HPC applications. We provide a succinct look into the inner technical workings of each component that should be instructive to both novices to HPC applications as well as to those who may be familiar with one component, but not necessarily the others, in the design and functioning of the total interconnect. A supercomputer interconnect is not a monolithic structure, and this tutorial aims to give non-experts a “big-picture” overview of its substructure with an appreciation of how and why features in one component influence those in others. We believe this is one of the first tutorials to discuss these three major components as one integrated whole. In addition, we give detailed examples of practical experience and typical algorithms used within each component in order to give insights into what issues and trade-offs are important.

NERSC has partnered with 20 representative application teams to evaluate performance on the Xeon-Phi Knights Landing architecture and develop an application-optimization strategy for the greater NERSC workload on the recently installed Cori system. In this article, we present early case studies and summarized results from a subset of the 20 applications highlighting the impact of important architecture differences between the Xeon-Phi and traditional Xeon processors. We summarize the status of the applications and describe the greater optimization strategy that has formed.

In the Community Earth System Model (CESM), the ocean model is computationally expensive for high-resolution grids and is often the least scalable component for high-resolution production experiments. The major bottleneck is that the barotropic solver scales poorly at high core counts. We design a new barotropic solver to accelerate the high-resolution ocean simulation. The novel solver adopts a Chebyshev-type iterative method to reduce the global communication cost in conjunction with an effective block pre-conditioner to further reduce the iterations. The algorithm and its computational complexity are theoretically analyzed and compared with other existing methods. We confirm the significant reduction of the global communication time with a competitive convergence rate using a series of idealized tests. Numerical experiments using the CESM 0.1 degrees global ocean model show that the proposed approach results in a factor of 1.7 speed-up over the original method with no loss of accuracy, achieving 10.5 simulated years per wall-clock day on 16 875 cores.

This project aimed to improve long term global climate simulations by resolving and enhancing the representation of the processes involved in the cycling of freshwater through estuaries and coastal regions. This was a collaborative multi-institution project consisting of physical oceanographers, climate model developers, and computational scientists. It specifically targeted the DOE objectives of advancing simulation and predictive capability of climate models through improvements in resolution and physical process representation.

While the InfiniBand link-by-link flow control helps avoid packet loss, it unfortunately causes the effects of congestion to spread through a network. Flows whose paths do not even pass through congested ports could suffer from reduced throughput. We propose a Dynamic Congestion Management System DCMS to address this problem. Without per-flow information, the DCMS leverages performance counters of switch ports to detect onset of congestion, and determines whether-or-not victim flows are present. The DCMS then takes actions to cause an aggressive reduction in the sending rates of congestion-causing contributor flows if victim flows are present. On the other hand, in the absence of victim flows, the DCMS allows the contributor flows to maintain high sending rates and finish as quickly as possible. Our results show that dynamic congestion management can enable a network to serve both contributor flows and victim flows effectively. The DCMS solution operates within the constraints of the InfiniBand Standard.