We introduce the Artificial Intelligence Forecasting System for Direct Observation Prediction (AIFS-DOP). AIFS-DOP is trained on a 40-year harmonized dataset of gridded observations, without using numerical weather prediction (NWP) reanalysis or model data. The resulting model is competitive with ECMWF's Integrated Forecasting System (IFS) when scored on a one year period of forecasts across 2021/2022. This progress on Direct Observation Prediction represents the first time that a data-driven model, trained solely on observations, is competitive with the IFS at medium ranges for several key upper-air and surface headline scores, when verified against observation data.

Earth system reanalysis datasets are foundational for weather and climate research and provide the gridded training data used by most machine learning weather prediction systems. Here we show results from a prototype system that suggest that machine learning models trained only on Earth system observations can potentially be used to generate multi-decade global reanalyses without using physics-based numerical models. The resulting gridded fields capture large-scale atmospheric structure and variability across multiple timescales, while exhibiting signs of physical coherence in several key dynamical diagnostics. Evaluations of the prototype against held-out independent atmospheric observations indicate that the root mean square vector error of upper-level winds is close to that of ERA5 when compared at a consistent resolution, and that the standard deviation of the error at the surface is between that of 4th- and 5th-generation ECMWF reanalyses (ERA-Interim and ERA5). Furthermore, while traditional reanalysis production is computationally expensive, typically taking several years to produce, the reanalysis presented here was generated during the course of a single working day. These results suggest that observation-trained machine learning models offer a promising new approach for reanalysis production from observations alone.

Abstract Data assimilation (DA) combines observations with numerical models to estimate evolving Earth system states for forecasting and monitoring. Machine learning (ML) enables surrogate modeling, pattern recognition and Bayesian inference. These fields are converging: ML accelerates DA, while DA provides uncertainty quantification and physical constraints. Hybrid DA-ML systems are promising, yet challenges persist in generalization, consistency and reproducibility. These approaches are increasingly integrated, shaping next-generation prediction systems and observing networks.

The Data Assimilation (DA) community has been developing various diagnostics to understand the importance of the observing system in accurately forecasting the weather. They usually rely on the ability to compute the derivatives of the physical model output with respect to its initial condition. For example, the Forecast Sensitivity-based Observation Impact (FSOI) estimates the impact on the forecast error of each observation processed in the DA system. This paper presents how these DA diagnostic tools are transferred to Machine Learning (ML) models, as their derivatives are readily available through automatic differentiation. We specifically explore the interpretability and explainability of the observation-driven GraphDOP model developed at the European Centre for Medium-Range Weather Forecasts (ECMWF). The interpretability study demonstrates the effectiveness of GraphDOP's sliding attention window to learn the meteorological features present in the observation datasets and to learn the spatial relationships between different regions. Making these relationships more transparent confirms that GraphDOP captures real, physically meaningful processes, such as the movement of storm systems. The explainability of GraphDOP is explored by applying the FSOI tool to study the impact of the different observations on the forecast error. This inspection reveals that GraphDOP creates an internal representation of the Earth system by combining the information from conventional and satellite observations.

Through a series of experiments, we provide evidence that the GraphDOP model - trained solely on meteorological observations, using no prior knowledge - develops internal representations of the Earth System state, structure and dynamics as well as the characteristics of different observing systems. Firstly, we demonstrate that the network constructs a unified latent representation of the Earth System state which is common across different observation types. For example, cloud structures maintain physical consistency whether viewed in predictions for satellite radiances from different sensors, or for direct in-situ measurements of the cloud fraction. Secondly, we show examples that suggest that the network learns to emulate viewing effects - learned observation operators that map from the unified state representation to observed properties. Microwave sounder limb effects and geometric viewing effects, such as sunglint in visible imagery, are both well captured. Finally, we demonstrate that the model develops rich internal representations of the structure of meteorological systems and their dynamics. For instance, when the network is only provided with observations from a single infrared instrument, it is able to infer unobserved, non-local structures such as jet streams, surface pressure patterns and warm and cold air masses associated with synoptic systems. This work provides insights into how neural networks trained solely on observations of the Earth System spontaneously develop coherent internal representations of the physical world in order to meet the training objective - enhancing our understanding and guiding future development of these models.

Interactions between different components of the Earth System (e.g. ocean, atmosphere, land and cryosphere) are a crucial driver of global weather patterns. Modern Numerical Weather Prediction (NWP) systems typically run separate models of the different components, explicitly coupled across their interfaces to additionally model exchanges between the different components. Accurately representing these coupled interactions remains a major scientific and technical challenge of weather forecasting. GraphDOP is a graph-based machine learning model that learns to forecast weather directly from raw satellite and in-situ observations, without reliance on reanalysis products or traditional physics-based NWP models. GraphDOP simultaneously embeds information from diverse observation sources spanning the full Earth system into a shared latent space. This enables predictions that implicitly capture cross-domain interactions in a single model without the need for any explicit coupling. Here we present a selection of case studies which illustrate the capability of GraphDOP to forecast events where coupled processes play a particularly key role. These include rapid sea-ice freezing in the Arctic, mixing-induced ocean surface cooling during Hurricane Ian and the severe European heat wave of 2022. The results suggest that learning directly from Earth System observations can successfully characterise and propagate cross-component interactions, offering a promising path towards physically consistent end-to-end data-driven Earth System prediction with a single model.

State-of-the-art data assimilation systems, such as the 4DVar system of the European Centre for Medium-Range Weather Forecasts (ECMWF), are highly successful in producing state estimates of the atmosphere constrained by millions of observations. However, existing systems require substantial approximations, e.g. in forward operators, and employ conventional models in their optimization loop. This limits the amount of information that can be extracted from observations, e.g. the assimilation of visible satellite channels is still challenging. The current approach also separates the use of observations into a data assimilation step and a forecasting one, with observations only being used indirectly for forecasting, for example for tuning of parametrizations and for evaluation. Here, we explore the possibility to train large machine learning models for data assimilation and forecasting directly from observations. In particular, we build a generative transformer neural network that models the joint the probability distribution p(y,x) over output states y for an input x. The input are observations from a temporal window, e.g. 6h or 12h, and the output y can either be an estimate of the state within the window or a short-term forecast, e.g. for another 12h. Different observations are processed by different embedding networks but then fused in the backbone transformer network. To obtain an integrated and consistent representation of the atmospheric state that corresponds to the different input data streams, we train with a variation of the masked token model training objective from natural language processing that impels the network to learn the correlation between the different input streams and channels. To properly represent the statistical nature of the estimation of y given x, our network provides an ensemble prediction as a nonparametric model for the probability distribution over y. We present results for a network trained with a substantial amount of data, including different satellite observations (such as AMSU-A microwave sounders from NOAA 15-19 and the METOP satellites as well as IASI), radiosondes, and ground station-based measurements. The skill for both data assimilation and forecasting is analyzed and compared to ECMWF’s operational 4DVar system. We also ablate the effect different observations have on the skill of the network output.

We introduce GraphDOP, a new data-driven, end-to-end forecast system developed at the European Centre for Medium-Range Weather Forecasts (ECMWF) that is trained and initialised exclusively from Earth System observations, with no physics-based (re)analysis inputs or feedbacks. GraphDOP learns the correlations between observed quantities - such as brightness temperatures from polar orbiters and geostationary satellites - and geophysical quantities of interest (that are measured by conventional observations), to form a coherent latent representation of Earth System state dynamics and physical processes, and is capable of producing skilful predictions of relevant weather parameters up to five days into the future.

Skilful Machine Learned weather forecasts have challenged our approach to numerical weather prediction, demonstrating competitive performance compared to traditional physics-based approaches. Data-driven systems have been trained to forecast future weather by learning from long historical records of past weather such as the ECMWF ERA5. These datasets have been made freely available to the wider research community, including the commercial sector, which has been a major factor in the rapid rise of ML forecast systems and the levels of accuracy they have achieved. However, historical reanalyses used for training and real-time analyses used for initial conditions are produced by data assimilation, an optimal blending of observations with a physics-based forecast model. As such, many ML forecast systems have an implicit and unquantified dependence on the physics-based models they seek to challenge. Here we propose a new approach, training a neural network to predict future weather purely from historical observations with no dependence on reanalyses. We use raw observations to initialise a model of the atmosphere (in observation space) learned directly from the observations themselves. Forecasts of crucial weather parameters (such as surface temperature and wind) are obtained by predicting weather parameter observations (e.g. SYNOP surface data) at future times and arbitrary locations. We present preliminary results on forecasting observations 12-hours into the future. These already demonstrate successful learning of time evolutions of the physical processes captured in real observations. We argue that this new approach, by staying purely in observation space, avoids many of the challenges of traditional data assimilation, can exploit a wider range of observations and is readily expanded to simultaneous forecasting of the full Earth system (atmosphere, land, ocean and composition).

Schematic representation of the way incremental 4D-Var solves the nonlinear minimisation problem. The full nonlinear cost function (thick blue line) is minimised through the minimisation of successive quadratic cost functions (thin black lines) obtained by linearisation of the cost function around successively more accurate guess fields (Plot courtesy of Sebastien Massart, European Centre for Medium-Range Weather Forecasts).

This paper discusses the role of the observation operator in data assimilation and how the evolving strategy of Numerical Weather Prediction is creating new challenges for Radiative Transfer Model developments, placing an increasing demand on these models and their consistency, bias characterisation and range of capabilities. The paper concludes that to meet future challenges models will need to be developed in a modular style taking elements from many research groups, giving attention to consistency of physical assumptions, interfaces and coding standards.

1) p1L15: Rabier and Courtier (QJRMS 1992) presented a good study to measure the accuracy of the tangent-linear with a "realistic" global baroclinic model, the IFS of the time but without the physical parameterizations. Lacarra and Talagrand (1988) did also conclude that the TL model was a reasonable approximation of the evolution of a perturbation for ~48-h. This is the basis of ECMWF's EPS.P4, line 24: “Holm et al., 2003 should be Holm et al. 2002. “.

The ability of a data assimilation system to deal effectively with nonlinearities arising from the prognostic model or the relationship between the control variables and the available observations has received a lot of attention in theoretical studies based on very simplified test models. Less work has been done to quantify the importance of nonlinearities in operational, state-of-the-art global data assimilation systems. In this paper we analyse the nonlinear effects present in ECMWF 4D-Var and evaluate the ability of the incremental formulation to solve the nonlinear assimilation problem in a realistic NWP environment. We find that nonlinearities have increased over the years due to a combination of increased model resolution and the ever-growing importance of observations that are nonlinearly related to the state. Incremental 4D-Var is well suited for dealing with these nonlinear effects, but at the cost of increasing the number of outer loop relinearisations. We then discuss strategies for accommodating the increasing number of sequential outer loops in the tight schedules of operational global NWP.

Data-intensive software is increasingly prominent in today’s world, where the collection, processing, and dissemination of ever-larger volumes of data has become a driving force behind innovation in the early twenty-first century. The trend towards massive data manipulation is broad-based, and case studies can be examined in domains from politics, to intelligence gathering, to scientific and medical research. The scientific domain in particular provides a rich array of case studies that offer ready insight into many of the modern software engineering, and software architecture challenges associated with data-intensive systems.

Atmospheric motion vectors (AMVs) have been produced for decades and remain an important source of wind information. Many studies have suggested that the traditional interpretation of AMVs as representative of the wind at cloud top is suboptimal and that they are more representative of the winds within the cloud. This paper investigates the vertical representativity of cloudy AMVs using both first-guess departure [observation - background (O - B)] statistics and the simulation-study technique. A state-of-the-art convection-permitting mesoscale model ("UKV") is used in conjunction with a radiative transfer model and the Nowcasting Satellite Application Facility (NWCSAF) AMV package to produce synthetic AMVs over a 1-month period. The simulated upper-level AMVs suffered from large height-assignment errors uncharacteristic of those in reality; these issues were partially alleviated by using the model cloud top instead of the assigned height. In agreement with previous studies, both the simulated and real AMVs were found to have the closest fit to a layer mean of the model winds with the majority of the layer below the estimated cloud top. However, improvements in the fit between the AMVs and the model were also found by simply lowering the assigned height. A short NWP trial hinted that height reassignment might lead to short-range forecast improvements. The results of this study indicate that the simulation technique was able to match the usefulness of O - B statistics for AMVs associated with low- and medium-level clouds (albeit at a higher computational cost); however, challenges remain in the simulation of upper-level clouds.

The vertical representivity of Atmospheric Motion Vectors (AMVs) has been the subject of much discussion over many years. Several recent studies have suggested that the traditional interpretation of AMVs as representative of the wind at cloud top is sub-optimal and that they are in fact more representative of the winds within the cloud. In this study, the vertical representivity of cloudy AMVs is investigated using both real and simulated AMVs. A state of the art convection permitting mesoscale model is used in conjuction with the RTTOV radiative transfer scheme and the Nowcasting SAF (NWCSAF) AMV package to produce synthetic AMVs over a one month period, which are then compared against the model truth from which they were derived. In agreement with previous studies, the results indicated that AMVs are more representative of a layer mean wind with the majority of the layer below the cloud top. This suggests that the use of a layer averaging observation operator may benefit the assimilation of AMVs into NWP models. Improvements in the fit between the AMVs and the model can also be found by simply lowering the assigned height by around 40hPa. The use of either a lower assigned height or a layer average significantly reduced the slow bias commonly seen in upper level AMVs. The utility of the simulation study data in comparison to traditional O-B statistics was assessed. The results for low and medium height clouds were encouraging with the simulated AMV error characteristics similar to those in real AMVs. However, the simulated upper level AMVs suffered from significantly larger height assignment errors than those in reality making the results unrepresentative.

A simulation study has been conducted to characterise, quantify and understand errors in Atmospheric Motion Vector (AMV) wind retrievals within a perfect model framework. Synthetic model radiances are generated by passing model data from the Met Office ‘UKV’ 1.5km grid length model into the RTTOV radiative transfer model. Synthetic AMVs are then generated from these model radiances using the NWCSAF high resolution wind package. The perfect model framework allows errors in the AMV wind retrievals to be quantified as the ‘true’ model winds are known at every grid point. All current AMV algorithms assume that clouds act as passive tracers drifting with the wind and that sources and sinks are negligible. Results are presented of an experiment to quantify the extent to which clouds act as passive tracers as a function of time and spatial scale. By configuring the model to turn off all cloud sources and sinks, it is shown that although the passive tracer assumption errors are small in this case study, sources and sinks of cloud become more significant with increasing time, and become less significant as the spatial scale is increased. The implications of these results for appropriate configuration of AMV tracking box size and time interval are discussed.

The assimilation of satellite derived Atmospheric Motion Vectors (AMVs) into numerical weather prediction (NWP) models have provided benefits to operational forecasts for many years. Comparisons of AMVs against sonde observations are routinely used to monitor the accuracy of the AMVs. Simulation studies, whereby AMVs are derived from model cloud fields, provide a useful framework to characterise and understand the sources of errors in AMV wind estimates. The latest generation of high resolution NWP models with grid lengths of order 1km are able to resolve most convective cloud features explicitly. These models have been shown to realistically simulate the life cycle and motion of many cloud features. In this study, the Met Office ‘UKV’ mesoscale model is used to generate synthetic AMVs. Comparisons are presented between the synthetic AMVs and the ‘true’ model winds. The results confirm that the simulated AMVs produced by the system are realistic and provide a good framework for further studies.

The disparate communities of climate modeling and remote sensing are finding economic, political, and societal benefit from the direct comparisons of climate model outputs to satellite observations, using these comparisons to help tune models and to provide ground truth in understanding the Earth's climate processes. In the context of the Intergovernmental Panel on Climate Change (IPCC) and its upcoming 5th Assessment Report (AR5), the authors have been working with principals in both communities to build a software infrastructure that enables these comparisons. This infrastructure must overcome several software engineering challenges, including bridging heterogeneous data file formats and metadata formats, transforming swath-based remotely sensed data into globally gridded datasets, and navigating and aggregating information from the largely distributed ecosystem of organizations that house these climate model outputs and satellite data. The authors' focus in this article is on the description of software tools and services that meet these stringent challenges, and on informing the broader communities of climate modelers, remote sensing experts, and software engineers on the lessons learned from their experience so that future systems can benefit and improve upon their existing results.