Forecast verification is an essential function of National Meteorological and Hydrological Services (NMHSs), underpinning their ability to deliver accurate, reliable, and actionable weather, climate, and water-related information. As NMHSs face increasing demands for transparency, accountability, and continuous improvement, they require robust systems to assess and enhance the quality of their forecasts. This article presents a holistic forecast verification capability development framework, built from over a decade of focused effort at the Australian Bureau of Meteorology. The framework integrates best practices in governance, data management, verification metrics, and communication. It acknowledges the importance of user-centered approaches and highlights areas where verification practices can align with user needs. To support NMHSs in adopting this framework, the article introduces two practical tools: a Verification Planning Template for establishing new verification activities and systems and a Gap Analysis and Maturity Assessment (GAMA) tool for benchmarking and advancing existing practices. These tools provide structured guidance for planning, evaluating, and improving verification within a NMHS, with the ultimate goal of delivering higher quality forecasts that meet diverse stakeholder needs. The Bureau's progress in implementing this framework demonstrates significant benefits, including improved forecast quality, enhanced coordination across verification efforts, and greater trust among users. However, challenges such as data availability, system integration, and resourcing remain pervasive, both within the Bureau and globally. The tools and insights shared in this article offer a pathway for NMHSs to overcome these obstacles, enabling them to better respond to evolving user expectations and operational demands. This work highlights the value of fostering a strong verification culture, supported by collaboration and knowledge sharing across the international meteorological community. By applying the principles and tools presented here, and customizing them to their circumstances, NMHSs can advance toward resilient, evidence-based verification practices and capabilities that enhance forecast reliability and stakeholder confidence worldwide.

`scores` is a Python package containing mathematical functions for the verification, evaluation and optimisation of forecasts, predictions or models. It supports labelled n-dimensional (multidimensional) data, which is used in many scientific fields and in machine learning. At present, `scores` primarily supports the geoscience communities; in particular, the meteorological, climatological and oceanographic communities. `scores` not only includes common scores (e.g., Mean Absolute Error), it also includes novel scores not commonly found elsewhere (e.g., FIxed Risk Multicategorical (FIRM) score, Flip-Flop Index), complex scores (e.g., threshold-weighted continuous ranked probability score), and statistical tests (such as the Diebold Mariano test). It also contains isotonic regression which is becoming an increasingly important tool in forecast verification and can be used to generate stable reliability diagrams. Additionally, it provides pre-processing tools for preparing data for scores in a variety of formats including cumulative distribution functions (CDF). At the time of writing, `scores` includes over 50 metrics, statistical techniques and data processing tools. All of the scores and statistical techniques in this package have undergone a thorough scientific and software review. Every score has a companion Jupyter Notebook tutorial that demonstrates its use in practice. `scores` supports `xarray` datatypes, allowing it to work with Earth system data in a range of formats including NetCDF4, HDF5, Zarr and GRIB among others. `scores` uses Dask for scaling and performance. Support for `pandas` is being introduced. The `scores` software repository can be found at https://github.com/nci/scores/

University students can learn about weather warnings and contribute to a database for the World Meteorological Organization (WMO) project on value chain approaches to evaluate the end-to-end warning chain. The project offers students a way to understand how information about high-impact weather is created, shared, and used within a complete warning system for a selected event. Their contributions are intended to inform researchers and practitioners on what has and what has not worked well in the warning process. The students use a structured questionnaire designed to collect information on observations, forecasting, hazards, impacts, warning communications, and responses.

Operational agencies face significant challenges related to the verification and evaluation of weather forecasts. These challenges were investigated in a series of online workshops and polls engaging operational personnel from six countries. Five key themes emerged: inadequate verification approaches for both existing and emerging products; incomplete and uncertain observations; difficulties in accurately capturing users’ real-world experiences using simplified metrics; poor communication and understanding of forecasts and complex verification information; and institutional factors such as limited resources, evolving meteorologist roles, and concerns over reputational damage. We identify nearly 50 operationally relevant scientific questions and suggest calls to action. Addressing these needs includes designing forecast systems with verification as a central consideration, enhancing the availability of observations, and developing and adopting community software systems. Additionally, we propose the establishment of an international community comprising environmental and social science researchers, statisticians, verification practitioners, and users to provide sustained support for this collective endeavor.

"Enhancing the Value of Weather and Climate Services in Society: Identified Gaps and Needs as Outcomes of the First WMO WWRP/SERA Weather and Society Conference" published on 17 Mar 2023 by American Meteorological Society.

The weather information value chain provides a framework for characterising the production, communication, and use of information by all stakeholders in an end-to-end warning system covering weather and hazard monitoring, modelling and forecasting, risk assessment, communication and preparedness activities. Warning services are typically developed and provided through a multitude of complex and malleable value chains (networks), often established through co-design, co-creation and co-provision. In November 2020, a 4-year international project under the World Meteorological Organization (WMO) World Weather Research Programme was instigated to explore value chain approaches to describe and evaluate warning systems for high impact weather by integrating physical and social science. It aims to create a framework with guidance and tools for using value chain approaches, and to develop a database of high impact weather warning case studies for scientists and practitioners to review, analyse and learn from previous experience using value chain approaches. Here we describe a template for high-impact weather event case study collection that provides a tool for scientists and practitioners involved in researching, designing and evaluating weather-related warning systems to review previous experience of high impact weather events and assess their efficacy.

Inhalation of grass pollen can result in acute exacerbation of asthma, prompting questions about how grass pollen reaches metropolitan areas. We establish typical atmospheric Poaceae (grass) pollen concentrations recorded at two pollen samplers within the Sydney basin in eastern Australia and analyse their correlation with each other and meteorological variables. We determine the effect of synoptic and regional airflow on Poaceae pollen transport during a period of extreme (≥ 100 grains m −3 air) concentration and characterise the meteorology. Finally, we tested the hypothesis that most Poaceae pollen captured by the pollen samplers originated from local sources. Fifteen months of daily pollen data, three days of hourly atmospheric Poaceae pollen concentrations and fifteen months of hourly meteorology from two locations within the Sydney basin were used. Weather Research Forecasting (WRF), Hybrid Single-Particle Lagrangian Integrated Trajectory (HYSPLIT) modelling and conditional bivariate probability functions (CBPF) were used to assess Poaceae pollen transport. Most Poaceae pollen collected was estimated to be from local sources under low wind speeds. Extreme daily Poaceae pollen concentrations were rare, and there was no strong evidence to support long-distance Poaceae pollen transport into the Sydney basin or across the greater Sydney metropolitan area. Daily average pollen concentrations mask sudden increases in atmospheric Poaceae pollen, which may put a significant and sudden strain on the healthcare system. Mapping of Poaceae pollen sources within Sydney and accurate prediction of pollen concentrations are the first steps to an advanced warning system necessary to pre-empt the healthcare resources needed during pollen season.

The International Verification Methods Workshop was held online in November 2020 and included sessions on physical error characterization using process diagnostics and error tracking techniques; exploitation of data assimilation techniques in verification practices, e.g., to address representativeness issues and observation uncertainty; spatial verification methods and the Model Evaluation Tools, as unified reference verification software; and meta-verification and best practices for scores computation. The workshop reached out to diverse research communities working in the areas of high-impact weather, subseasonal to seasonal prediction, polar prediction, and sea ice and ocean prediction. This article summarizes the major outcomes of the workshop and outlines future strategic directions for verification research.

Epidemic asthma events represent a significant risk to emergency services as well as the wider community. In southeastern Australia, these events occur in conjunction with relatively high amounts of grass pollen during the late spring and early summer, which may become concentrated in populated areas through atmospheric convergence caused by a number of physical mechanisms including thunderstorm outflow. Thunderstorm forecasts are therefore important for identifying epidemic asthma risk factors. However, the representation of thunderstorm environments using regional numerical weather prediction models, which are a key aspect of the construction of these forecasts, have not yet been systematically evaluated in the context of epidemic asthma events. Here, we evaluate diagnostics of thunderstorm environments from historical simulations of weather conditions in the vicinity of Melbourne, Australia, in relation to the identification of epidemic asthma cases based on hospital data from a set of controls. Skillful identification of epidemic asthma cases is achieved using a thunderstorm diagnostic that describes near-surface water vapor mixing ratio. This diagnostic is then used to gain insights on the variability of meteorological environments related to epidemic asthma in this region, including diurnal variations, long-term trends, and the relationship with large-scale climate drivers. Results suggest that there has been a long-term increase in days with high water vapor mixing ratio during the grass pollen season, with large-scale climate drivers having a limited influence on these conditions. SIGNIFICANCE STATEMENT: We investigate the atmospheric conditions associated with epidemic thunderstorm asthma events in Melbourne, Australia, using historical model simulations of the weather. Conditions appear to be associated with high atmospheric moisture content, which relates to environments favorable for severe thunderstorms, but also potentially pollen rupturing as suggested by previous studies. These conditions are shown to be just as important as the concentration of grass pollen for a set of epidemic thunderstorm asthma events in this region. This means that weather model simulations of thunderstorm conditions can be incorporated into the forecasting process for epidemic asthma in Melbourne, Australia. We also investigate long-term variability in atmospheric conditions associated with severe thunderstorms, including relationships with the large-scale climate and long-term trends.

When many people develop asthma symptoms over a short period of time because of a combination of high concentrations of airborne allergens and strong thunderstorm outflows, this is known as epidemic thunderstorm asthma (ETSA).Worldwide, at least 22 ETSA events have been reported since 1983, with 10 occurring in southeastern Australia during late spring or early summer, when temperate grasses are flowering and there are high levels of airborne grass pollen (some of which may be ruptured into tiny allergenic starch granules and concentrated near ground level where they can affect the lower airways of susceptible people).Events have also been reported in the United Kingdom and other European countries, the Middle East, the United States, and Canada.

In November 2016, an unprecedented epidemic thunderstorm asthma event in Victoria, Australia, resulted in many thousands of people developing breathing difficulties in a very short period of time, including 10 deaths, and created extreme demand across the Victorian health services. To better prepare for future events, a pilot forecasting system for epidemic thunderstorm asthma (ETSA) risk has been developed for Victoria. The system uses a categorical risk-based approach, combining operational forecasting of gusty winds in severe thunderstorms with statistical forecasts of high ambient grass pollen concentrations, which together generate the risk of epidemic thunderstorm asthma. This pilot system provides the first routine daily epidemic thunderstorm asthma risk forecasting service in the world that covers a wide area, and integrates into the health, ambulance, and emergency management sector. Epidemic thunderstorm asthma events have historically occurred infrequently, and no event of similar magnitude has impacted the Victorian health system since. However, during the first three years of the pilot, 2017–19, two high asthma presentation events and four moderate asthma presentation events were identified from public hospital emergency department records. The ETSA risk forecasts showed skill in discriminating between days with and without health impacts. However, even with hindsight of the actual weather and airborne grass pollen conditions, some high asthma presentation events occurred in districts that were assessed as low risk for ETSA, reflecting the challenge of predicting this unusual phenomenon.

This study demonstrates the useful information that can be derived from contiguous rain area (CRA) evaluation, such as systematic errors in tropical cyclone (TC) rainfall location and components of rainfall error due to incorrect predictions of location, rain volume, and rain pattern. CRA verification uses pattern matching techniques to determine the location error, as well as errors in area, mean and maximum intensity, and spatial pattern. In this study, CRA verification was applied to evaluate Australian Community Climate and Earth System Simulator (ACCESS)‐TC, the TC version of ACCESS, daily rainfall forecasts over 15 TCs in the north west Pacific ocean during 2012–2013, by comparing with Tropical Rainfall Measuring Mission (TRMM) 3B42 satellite estimates. The results showed that pattern error was the major contributor to the total TC rainfall forecast error, followed by volume and displacement. ACCESS‐TC forecasts tended to predict more rainfall closer to the TC center compared to Tropical Rainfall Measuring Mission (TRMM) 3B42 estimates. This bias occurred for different CRA rainfall thresholds, verification grid resolutions and forecast lead times. Furthermore, rain event verification showed that for short lead time (24 hr) forecasts, overestimation of rain volume was a major problem for ACCESS‐TC forecasts, while displacement error was more significant in longer lead time (72 hr) forecasts. Finally, we compared empirical probability distribution functions and radial probability distributions of rainfall in the forecasts and observations to further characterise the rain volume error. This confirmed that ACCESS‐TC tended to produce more extreme rain in the locations closer to the TC center (eyewall).

Recent advancements in numerical weather prediction (NWP) and the enhancement of model resolution have created the need for more robust and informative verification methods. In response to these needs, a plethora of spatial verification approaches have been developed in the past two decades. A spatial verification method intercomparison was established in 2007 with the aim of gaining a better understanding of the abilities of the new spatial verification methods to diagnose different types of forecast errors. The project focused on prescribed errors for quantitative precipitation forecasts over the central United States. The intercomparison led to a classification of spatial verification methods and a cataloging of their diagnostic capabilities, providing useful guidance to end users, model developers, and verification scientists. A decade later, NWP systems have continued to increase in resolution, including advances in high-resolution ensembles. This article describes the setup of a second phase of the verification intercomparison, called the Mesoscale Verification Intercomparison over Complex Terrain (MesoVICT). MesoVICT focuses on the application, capability, and enhancement of spatial verification methods to deterministic and ensemble forecasts of precipitation, wind, and temperature over complex terrain. Importantly, this phase also explores the issue of analysis uncertainty through the use of an ensemble of meteorological analyses.

Forecasts of precipitation produced by global and regional versions of the Bureau of Meteorology’s Australian Community Climate and Earth System Simulator (ACCESS) numerical weather prediction models are compared with Tropical Rainfall Measuring Mission (TRMM) observations for the period January 2011 to March 2014. The area considered covers longitudes 110° to 160°E within 40° latitude of the equator, and includes the Australian continent and part of south-east Asia. Forecast accuracy is assessed using objective measures: equitable threat score (ETS), frequency bias (FB), and the ratio of predicted to observed rain volume (RVR). For the assessment, the TRMM and model datasets are both interpolated to consistent grids defined by spacings of 1° longitude and 1° latitude. Assessments based on three-month seasons show that, in general, the volume of rain predicted by the models is too high, and rainfall is predicted to occur at more locations than observed. Latitudinal variations in values of the ETS reveal marked declines over the equatorial tropics, with minimum values near the equator from December to May and near 10°N from June to November. Differences in the ETS between 24-hour model and persistence forecasts show that the models produce useful predictions of rainfall away from the tropics, poleward of latitudes 5º-15º S and 15º-30º N, depending on the season and model. The 48-hour model predictions are more useful than the 24-hour forecasts, with improvements relative to persistence over most latitudes, although differences are close to zero at low latitudes. Varying the rain threshold used to compute skill metrics shows that the models produce excessive rainfall at low rain rates, generally less than approximately 15 mm day-1, while not enough precipitation is forecast at higher rain rates. Values of the ETS are generally highest at lower rain rates, coinciding with the excessive values of RVR and FB.

Grain yields vary widely between seasons in rain-fed agriculture. The yield variability is strongly influenced by rainfall variability and a number of related crop management decisions. This is well recognised in the literature through the use of seasonal rainfall forecasts applied to the main cropping decisions. However, the value of short-term 10-day rainfall forecasts for management decisions in cropping has not yet been quantified. Here we report on the potential benefits of a hypothetical, always-correct 10-day rainfall (greater than 10 mm in three days) forecast used to determine early and late in-season crop management decisions.Most of the analysed applications of short-term rainfall forecasts show a significant increase in cropping profitability depending on rainfall region and soil type. Using a 10-day rainfall forecast to dry-sow prior to the traditional start of sowing at the first autumn rainfall can yield an extra A$20,000 to A$200,000 for a typical farm (i.e. A$10 to A$100/ha for a 2000 ha cropping programme). The same forecast type can be used to determine late in-season decisions on N fertiliser, and fungicide applications to control rust at the end-of-ear growth stage. In the one-third of seasons with late rainfall, the increased yield or decreased fungal damage can lead to benefits of A$10 to A$160/ha. When 10-day rainfall forecasts are applied together within a season, the extra benefits from correct short-term forecasts can be cumulative.Fine-tuning the forecast length, rainfall thresholds and exploring other possible crop decisions could lead to further increased returns in cropping from short-term rainfall forecasts. Ultimately, using hind-casts of short-term rainfall forecasts to measure the forecast skill and the implication of occasional non-correct forecasts will determine the actual value of such forecasts. (C) 2015 Elsevier B.V. All rights reserved.

We present a numerical investigation of a non-forecast sea-breeze-initiated thunderstorm that occurred unexpectedly in the eastern complex area of the Iberian Peninsula (Spain) on 7 August 2008. A high horizontal (2.5-km grid spacing) and vertical (60 sigma levels) resolution set-up of the hydrostatic HIRLAM model is used to simulate the evolution of isolated convection associated with sea breezes. The convective inhibition, convective available potential energy, vertical velocity, wind and precipitation fields are examined here in order to analyze the role of low-level sea-breeze convergence and sea-breeze front development in initiating intense convective activity (heavy rainfall, hail and gusty winds) under weakly defined synoptic disturbances. We observe that convective inhibition layers are eroded by sea-breeze frontal zones, increasing the convective available potential energy due to the enhanced vertical motion, forcing parcels to lift to the level of free convection. We show that the location of thunderstorms along the east coast of the Iberian Peninsula is partly controlled by the impact of sea-breeze fronts on modulating the diurnal spatio-temporal evolution of convective inhibition, convective available potential energy and updrafts.

In this study we investigated sea breeze thunderstorms with intense convective activity (i.e., heavy rainfall, hail and gusty winds) that occurred over the eastern Iberian Peninsula (Spain) and were missed by the operational HIRLAM model. We used two grid-spacing setups (5.0km and 2.5km) of the hydrostatic HIRLAM model, and the non-hydrostatic spectral HARMONIE suite (2.5km), to simulate isolated convection associated with sea breezes. The overall aim is to estimate the ability of these three experimental setups, in particular the HARMONIE model as the forthcoming operational numerical weather prediction in most European Weather Services, to correctly simulate convective precipitation associated with sea breezes. We evaluated high-resolution gridded precipitation forecasts from HIRLAM and HARMONIE suites for 15 sea breeze thunderstorms against high-density gridded raingauge measurements applying different neighborhood verification techniques. The results indicate that higher horizontal resolutions of HIRLAM and HARMONIE models succeeded in predicting the occurrence of these missed sea breeze thunderstorms, the HARMONIE suite being the most capable of providing good estimates of accumulated precipitation in convective events in terms of space and time. Advances in quantitative precipitation forecasting of locally driven convection could have practical applications for nowcasting dangerous sea breeze convective phenomena.