A Subjective and Objective Evaluation of Model Forecasts of Sumatra Squall Events

Sumatra squalls are important rain-bearing weather systems that affect Singapore and southern Peninsular Malaysia. The performance of forecasts for 63 past squall events is evaluated using a subjective evaluation by forecasters and an objective evaluation based on the fractions skill score (FSS). The purpose of this study is to investigate whether an objective procedure can reproduce the main results of the subjective evaluation. A convection permitting version of the Met Office (UKMO) Unified Model (UM), configured for a limited domain in the southern region of the South China Sea, is used with two driving global deterministic models: the UM and the European Centre for Medium-Range Weather Forecasts (ECMWF) model. Subjective and objective evaluation scoring methods for the two limited-area forecasts of the UM are compared, and it is shown that the objective procedure can reasonably emulate the scores produced by the forecasters in the context of parameters that are of direct relevance to the forecast process. This indicates that automated objective verification methods may be a reasonable alternative to resource intensive subjective evaluations for some cases. The robustness of the objective results is investigated using 7 months of data, and issues of statistical significance are considered.