The CF (Climate and Forecast) metadata conventions for netCDF datasets describe means of "compression-by-convention", i.e. methods for compressing and decompressing data according to algorithms that are fully described within the conventions themselves. These algorithms, which can be lossless or lossy, are not applicable to arbitrary data, rather the data have to exhibit certain characteristics to make the compression worthwhile, or even possible.Aggregation, available in CF-1.13, provides the utility of being able to view, as a single entity, a dataset that has been partitioned across multiple other independent datasets on disk, whilst taking up very little extra space on disk since the aggregation dataset contains no copies of the data in each component dataset. Aggregation can facilitate a range of activities such as data analysis, by avoiding the computational expense of deriving the aggregation at the time of analysis; archive curation, by acting as a metadata-rich archive index; and the post-processing of model simulation outputs, by spanning multiple datasets written at run time that together constitute a more cohesive and useful product. CF aggregation currently has cf-python and xarray implementations.The conceptual CF data model does not recognise compression nor aggregation, choosing to view all CF datasets as if they were uncompressed and containing all of their own data. As a result, the cf-python data analysis library, that is built exactly on the CF data model, also presents datasets lazily to the user in this manner, without decompressing or re-combining the data in memory until the user actually accesses the data, at which time it occurs automatically. This approach allows the user to interact with their data in an intuitive and efficient manner; and also removes the need for the user to have to assimilate large parts of the CF conventions and having to create their own code for dealing with the compression and aggregation algorithms.We will introduce compression by ragged arrays (as used by Discrete Sampling Geometry features, such as timeseries and trajectories) and dataset aggregation, with cf-python examples to demonstrate the ease of use that arises from using the CF data model interpretation of the data.

Abstract. The CMIP6 project was the most expansive and ambitious Model Intercomparison Project (MIP), the latest in a long history, extending back four decades. CMIP has captivated and engaged a broad, growing community focused on improving our climate understanding. It has anchored our ability to quantify and attribute the drivers and responses of the observed climate changes we are experiencing today. The project's profound impact has been achieved by combining the latest climate science and technology. This has enabled the production of latest-generation climate simulations and the dissemination of their output, which has seen increased community attention in every successive phase. The review emphasizes the pragmatics of progressively scaling up efforts, the evolution of how the MIPs were implemented, and the coordinated efforts to establish a minimal infrastructure to make that possible, most recently delivering CMIP6.

We review how the international modelling community, encompassing integrated assessment models, global and regional Earth system and climate models, and impact models, has worked together over the past few decades to advance understanding of Earth system change and its impacts on society and the environment and thereby support international climate policy. We go on to recommend a number of priority research areas for the coming decade, a timescale that encompasses a number of newly starting international modelling activities, as well as the IPCC Seventh Assessment Report (AR7) and the second UNFCCC Global Stocktake. Progress in these priority areas will significantly advance our understanding of Earth system change and its impacts, increasing the quality and utility of science support to climate policy.We emphasize the need for continued improvement in our understanding of, and ability to simulate, the coupled Earth system and the impacts of Earth system change. There is an urgent need to investigate plausible pathways and emission scenarios that realize the Paris climate targets - for example, pathways that overshoot 1.5 or 2 degrees C global warming, before returning to these levels at some later date. Earth system models need to be capable of thoroughly assessing such warming overshoots - in particular, the efficacy of mitigation measures, such as negative CO2 emissions, in reducing atmospheric CO2 and driving global cooling. An improved assessment of the long-term consequences of stabilizing climate at 1.5 or 2 degrees C above pre-industrial temperatures is also required. We recommend Earth system models run overshoot scenarios in CO2-emission mode to more fully represent coupled climate-carbon-cycle feedbacks and, wherever possible, interactively simulate other key Earth system phenomena at risk of rapid change during overshoot. Regional downscaling and impact models should use forcing data from these simulations, so impact and regional climate projections cover a more complete range of potential responses to a warming overshoot. An accurate simulation of the observed, historical record remains a fundamental requirement of models, as does accurate simulation of key metrics, such as the effective climate sensitivity and the transient climate response to cumulative carbon emissions. For adaptation, a key demand is improved guidance on potential changes in climate extremes and the modes of variability these extremes develop within. Such improvements will most likely be realized through a combination of increased model resolution, improvement of key model parameterizations, and enhanced representation of important Earth system processes, combined with targeted use of new artificial intelligence (AI) and machine learning (ML) techniques. We propose a deeper collaboration across such efforts over the coming decade.With respect to sampling future uncertainty, increased collaboration between approaches that emphasize large model ensembles and those focussed on statistical emulation is required. We recommend an increased focus on high-impact-low-likelihood (HILL) outcomes - in particular, the risk and consequences of exceeding critical tipping points during a warming overshoot and the potential impacts arising from this. For a comprehensive assessment of the impacts of Earth system change, including impacts arising directly as a result of climate mitigation actions, it is important that spatially detailed, disaggregated information used to generate future scenarios in integrated assessment models be available for use in impact models. Conversely, there is a need to develop methods that enable potential societal responses to projected Earth system change to be incorporated into scenario development.The new models, simulations, data, and scientific advances proposed in this article will not be possible without long-term development and maintenance of a robust, globally connected infrastructure ecosystem. This system must be easily accessible and useable by modelling communities across the world, allowing the global research community to be fully engaged in developing and delivering new scientific knowledge to support international climate policy.

The Coupled Model Intercomparison Project (CMIP) is one of the biggest international efforts aimed at better understanding the past, present, and future of climate changes in a multi-model context. A total of 21 model intercomparison projects (MIPs) were endorsed in its sixth phase (CMIP6), which included 190 different experiments that were used to simulate 40 000 years and produced around 40 PB of data in total. This paper presents the main findings obtained from the CPMIP (the Computational Performance Model Intercomparison Project), a collection of a common set of metrics, specifically designed for assessing climate model performance. These metrics were exclusively collected from the production runs of experiments used in CMIP6 and primarily from institutions within the IS-ENES3 consortium. The document presents the full set of CPMIP metrics per institution and experiment, including a detailed analysis and discussion of each of the measurements. During the analysis, we found a positive correlation between the core hours needed, the complexity of the models, and the resolution used. Likewise, we show that between 5 %–15 % of the execution cost is spent in the coupling between independent components, and it only gets worse by increasing the number of resources. From the data, it is clear that queue times have a great impact on the actual speed achieved and have a huge variability across different institutions, ranging from none to up to 78 % execution overhead. Furthermore, our evaluation shows that the estimated carbon footprint of running such big simulations within the IS-ENES3 consortium is 1692 t of CO2 equivalent. As a result of the collection, we contribute to the creation of a comprehensive database for future community reference, establishing a benchmark for evaluation and facilitating the multi-model, multi-platform comparisons crucial for understanding climate modelling performance. Given the diverse range of applications, configurations, and hardware utilised, further work is required for the standardisation and formulation of general rules. The paper concludes with recommendations for future exercises aimed at addressing the encountered challenges which will facilitate more collections of a similar nature.

The emergence of exascale computing and artificial intelligence offer tremendous potential to significantly advance Earth system prediction capabilities. However, enormous challenges must be overcome to adapt models and prediction systems to use these new technologies effectively. A 2022 WMO report on exascale computing recommends "urgency in dedicating efforts and attention to disruptions associated with evolving computing technologies that will be increasingly difficult to overcome, threatening continued advancements in weather and climate prediction capabilities." Further, the explosive growth in data from observations, model and ensemble output, and postprocessing threatens to overwhelm the ability to deliver timely, accurate, and precise information needed for decision-making. Artificial intelligence (AI) offers untapped opportunities to alter how models are developed, observations are processed, and predictions are analyzed and extracted for decision-making. Given the extraordinarily high cost of computing, growing complexity of prediction systems, and increasingly unmanageable amount of data being produced and consumed, these challenges are rapidly becoming too large for any single institution or country to handle. This paper describes key technical and budgetary challenges, identifies gaps and ways to address them, and makes a number of recommendations.

To manage Earth in the Anthropocene, new tools, new institutions, and new forms of international cooperation will be required. Earth Virtualization Engines is proposed as an international federation of centers of excellence to empower all people to respond to the immense and urgent challenges posed by climate change.

Tropical cyclones (TCs) are important phenomena, and understanding their behavior requires being able to detect their presence in simulations. Detection algorithms vary; here we compare a novel deep learning-based detection algorithm (TCDetect) with a state-of-the-art tracking system (TRACK) and an observational dataset (IBTrACS) to provide context for potential use in climate simulations. Previous work has shown that TCDetect has good recall, particularly for hurricane-strength events. The primary question addressed here is to what extent the structure of the systems plays a part in detection. To compare with observations of TCs, it is necessary to apply detection techniques to reanalysis. For this purpose, we use ERA-Interim, and a key part of the comparison is the recognition that ERA-Interim itself does not fully reflect the observations. Despite that limitation, both TCDetect and TRACK applied to ERA-Interim mostly agree with each other. Also, when considering only hurricane-strength TCs, TCDetect and TRACK correspond well to the TC observations from IBTrACS. Like TRACK, TCDetect has good recall for strong systems; however, it finds a significant number of false positives associated with weaker TCs (i.e., events detected as having hurricane strength but are weaker in reality) and extratropical storms. Because TCDetect was not trained to locate TCs, a post hoc method to perform comparisons was used. Although this method was not always successful, some success in matching tracks and events in physical space was also achieved. The analysis of matches suggested that the best results were found in the Northern Hemisphere and that in most regions the detections followed the same patterns in time no matter which detection method was used.

Tropical cyclones are high-impact weather events that have large human and economic effects, so it is important to be able to understand how their location, frequency, and structure might change in a future climate. Here, a lightweight deep learning model is presented that is intended for detecting the presence or absence of tropical cyclones during the execution of numerical simulations for use in an online data reduction method. This will help to avoid saving vast amounts of data for analysis after the simulation is complete. With run-time detection, it might be possible to reduce the need for some of the high-frequency high-resolution output that would otherwise be required. The model was trained on ERA-Interim reanalysis data from 1979 to 2017, and the training was concentrated on delivering the highest possible recall rate (successful detection of cyclones) while rejecting enough data to make a difference in outputs. When tested using data from the two subsequent years, the recall or probability of detection rate was 92%. The precision rate or success ratio obtained was that of 36%. For the desired data reduction application, if the desired target included all tropical cyclone events, even those that did not obtain hurricane-strength status, the effective precision was 85%. The recall rate and the area under curve for the precision–recall (AUC-PR) compare favorably with other methods of cyclone identification while using the smallest number of parameters for both training and inference.

<p>Managing huge volumes of data is a problem now, and will only become worse with the advent of exascale computing and next generation observational systems. An important recognition is that data needs to be more easily migrated between storage tiers. Here we present a new solution, the Near-Line Data store (NLDS), for managing data migration between user facing storage systems and tape by using an object storage cache.  NLDS builds on lessons learned from previous experience developing the ESIWACE funded Joint Data Migration App (JDMA) and deploying it at the Centre for Environmental Data Analysis (CEDA). <br> <br>CEDA currently has over 50PB of data stored on a range of disk based storage systems.  These systems are chosen on cost, power usage and accessibility via a network, and include three different types of POSIX disk and object storage. Tens of PB of additional data are also stored on tape. Each of these systems has different workflows, interfaces and latencies, causing difficulties for users.  </p><p>NLDS, developed with ESIWACE2 and other funding, is a multi-tiered storage solution using object storage as a front end to a tape library.  Users interact with NLDS via a HTTP API, with a Python library and command-line client provided to support both programmatic and interactive use.  Files transferred to NLDS are first written to the object storage, and a backup is made to tape.  When the object storage is approaching capacity, a set of policies is interrogated to determine which files will be removed from it.  Upon retrieving a file, NLDS may have to first transfer the file from tape to the object storage, if it has been deleted by the policies.  This implements a multi-tier of hot (disk), warm (object storage) and cold (tape) storage via a single interface. While systems like this are not novel, NLDS is open source, designed for ease of redeployment elsewhere, and for use from both local storage and remote sites. </p><p>NLDS is based around a microservice architecture, with a message exchange brokering communication between the microservices, the HTTP API and the storage solutions.  The system is deployed via Kubernetes, with each microservice in its own Docker container, allowing the number of services to be scaled up or down, depending on the current load of NLDS.  This provides a scalable, power efficient system while ensuring that no messages between microservices are lost.  OAuth is used to authenticate and authorise users via a pluggable authentication layer. The use of object storage as the front end to the tape allows both local and remote cloud-based services to access the data, via a URL, so long as the user has the required credentials. </p><p>NLDS is a a scalable solution to storing very large data for many users, with a user-friendly front end that is easily accessed via cloud computing. This talk will detail the architecture and discuss how the design meets the identified use cases.</p>

Participants of the Berlin Summit on Earth Virtualization Engines (EVEs) discussed ideas and concepts to improve our ability to cope with climate change. EVEs aim to provide interactive and accessible climate simulations and data for a wide range of users. They combine high-resolution physics-based models with machine learning techniques to improve the fidelity, efficiency, and interpretability of climate projections. At its core, EVEs offer a federated data layer that enables simple and fast access to exabyte-sized climate data through simple interfaces. In this article, we summarize the technical challenges and opportunities for developing EVEs, and argue that they are essential for addressing the consequences of climate change.

The land surface, hydrological, and groundwater modelling communities all have expertise in simulating the hydrological processes at play in the terrestrial component of the Earth system. However, these communities, and the wider Earth system modelling community, have largely remained distinct with limited collaboration between disciplines, hindering progress in the representation of hydrological processes in the land component of Earth system models (ESMs). In order to address key societal questions regarding the future availability of water resources and the intensity of extreme events such as floods and droughts in a changing climate, these communities must come together and build on the strengths of one another to produce next-generation land system models that are able to adequately simulate the terrestrial water cycle under change. The development of a common modelling infrastructure can contribute to stimulating cross-fertilisation by structuring and standardising the interactions. This paper presents such an infrastructure, a land system framework, which targets an intermediate level of complexity and constrains interfaces between components (and communities) and, in doing so, aims to facilitate an easier pipeline between the development of (sub-)community models and their integration, both for standalone use and for use in ESMs. This paper first outlines the conceptual design and technical capabilities of the framework; thereafter, its usage and useful characteristics are demonstrated through case studies. The main innovations presented here are (1) the interfacing constraints themselves; (2) the implementation in Python (the Unified Framework for Hydrology, unifhy); and (3) the demonstration of standalone use cases using the framework. The existing framework does not yet meet all our goals, in particular, of directly supporting integration into larger ESMs, so we conclude with the remaining limitations of the current framework and necessary future developments.

Large ensembles of climate-scale model simulations are key tools for assessing climate risks, separating internal variability from external forcing, and interpreting the observational record. Several modelling centres have produced such ensembles over the past years. Here we present early plans for the development of a new Large Ensemble based on the HadGEM3 (Hadley Centre Global Environment Model version 3) climate model. The initial plan envisages a 40-member ensemble spanning 150 years of historical/scenario climate (1950-2100) at a resolution of N216 (about 60 km) in the atmosphere and ¼° in the ocean. This initiative is part of the recently started UK NERC multi-centre project CANARI (Climate change in the Arctic-North Atlantic Region and Impacts on the UK). CANARI aims to advance understanding of the impacts on the UK arising from climate variability and change in the Arctic-North Atlantic region, with a focus on extreme weather and the potential for rapid, disruptive change. While we aim for the new Large Ensemble to become a resource for a wide range of applications, it will support addressing the CANARI science questions in particular. These questions are concerned with, for example, the (i) projected Arctic change and potential lower-latitude influences through atmospheric or oceanic pathways, (ii) the projected change in the large-scale (North Atlantic) ocean/atmosphere circulation, its drivers, and interaction with weather systems, and (iii) projected impacts on the UK arising from extreme weather (windstorms and flooding, blocking, heatwaves and droughts). This poster invites discussion with the community on all aspects of the design of the new Large Ensemble, and particularly seeks input regarding * the choice/number of experiments to follow (from CMIP6 Scenario MIP), * the initialisation strategy, and the degree to which slow (10 years and longer) variability, particularly in the ocean, should be sampled, and * the desired output.

Abstract With high levels of the Delta variant of COVID-19 circulating in England during September 2021, schools are set to reopen with few school-based non-pharmaceutical interventions (NPIs). In this paper, we present simulation results obtained from the individual-based model, JUNE, for English school opening after a prior vaccination campaign using an optimistic set of assumptions about vaccine efficacy and the likelihood of prior-reinfection. We take a scenario-based approach to modelling potential interventions to assess relative changes rather than real-world forecasts. Specifically, we assess the effects of vaccinating those aged 16-17, those aged 12-17, and not vaccinating children at all relative to only vaccinating the adult population, addressing what might have happened had the UK began teenage vaccinations earlier. Vaccinating children in the 12-15 age group would have had a significant impact on the course of the epidemic, saving thousands of lives overall in these simulations. In the absence of such a vaccination campaign our simulations show there could still be a significant positive impact on the epidemic (fewer cases, fewer deaths) by continuing NPI strategies in schools. Our analysis suggests that the best results in terms of lives saved are likely derived from a combination of the now planned vaccination campaign and NPIs in schools.

The last few decades have seen a range of advances in climate science and consequential policy initiatives at both national and international levels. These advances have been built on the back of progress in modelling and in part been enabled by the global data sharing initiative - the Earth System Grid Federation (ESGF) - which has underpinned recent phases of the World Climate Research Programme's Coupled Model Intercomparison Projects. The ESGF itself consists of data nodes deployed by individual modelling centres and a backbone of software development and services delivered by a few core institutions. Within Europe, along with some shared development of model components, these core ESGF software development and services are coordinated by the European Network on Earth System Modelling (ENES) and supported by the H2020 IS-ENES Phase 3 research infrastructure project. We provide an historical overview on advances in policy-relevant science, such as the Intergovernmental Panel for Climate Change (IPCC), that have been enabled by long-term underpinning development and funding of the ENES and ESGF infrastructure. We illustrate the recent shift of research funding from physical science objectives alone towards funding services to society (and the necessary underpinning research). We stress the potential dangers of underfunding research infrastructures that need to be simultaneously flexible and reliable enough to serve both ongoing basic research and the growing societal objectives, as emphasised by the development of climate services such as Copernicus Climate Change Service. We conclude by presenting some steps towards sustaining such research infrastructure in the context of the ENES and the possible futures of climate science.