A non-parametric way to estimate observation errors based on ensemble innovations

Previous studies that inferred the observation error statistics from the innovation statistics can only provide the second moment of the error probability density function (pdf). However, the observation errors are sometimes non-Gaussian, for example, for observation operators with unknown representation errors, or for bounded observations. In this study, we propose a new method, the Deconvolution-based Observation Error Estimation (DOEE), to infer the full observation error pdf. DOEE does not rely on linear assumptions on the observation operator, the optimality of the data assimilation algorithm, or implicit Gaussian assumptions on the error pdf. The main assumption of DOEE is the availability of an ensemble of background forecasts following the independent and identically distributed (i.i.d.) assumption. We conduct idealized experiments to demonstrate the ability of the DOEE to accurately retrieve a non-Gaussian (bimodal, skewed, or bounded) observation error pdf. We then apply the DOEE to construct a state-dependent observation error model for satellite radiances by stratifying the observation errors based on cloud amount. In general, we find that the observation error pdfs in many categories are skewed. By adding a new predictor, total column water vapor (TCWV), into the state-dependent model, we find that for cloudy pixels, when TCWV is small, the observation error pdfs are quite similar and Gaussian-like, whereas when TCWV is large, the observation error pdfs differ for different cloud amount, while all of them are positively biased. This result suggests that exploring other predictors, like cloud type, might improve the stratification of the observation error model. We also discuss ways to include a non-parametric observation error pdf into modern data assimilation schemes, including a consideration of the strong-constraint 4D-Var perspective, as well as the implications for other observation types including observations with bounded range. We propose a new method, the Deconvolution-based Observation Error Estimation (DOEE), to infer the non-parametric observation error pdf, without the linear assumption on the observation operator, the optimality of the data assimilation algorithm, or any implicit Gaussian assumptions on the errors. The main assumption of DOEE is the availability of an ensemble of background forecasts following the independent and identically distributed (i.i.d.) assumption. DOEE can retrieve non-parametric (bimodal, skewed) observation error pdfs accurately. image