Abstract. Spaceborne radar observations provide valuable information on the vertical structure of precipitation and hydrometeors, making them an important resource for numerical weather prediction and model evaluation. The Joint Effort for Data assimilation Integration (JEDI) is the unified and configurable data assimilation system developed and stewarded by the Joint Center for Satellite Data Assimilation (JCSDA) on behalf of its U.S. federal partners being adopted in NASA’s Goddard Earth Observing System (GEOS) suite of products. This study describes a new spaceborne radar data assimilation capability developed for use in JEDI, including both the radar forward operator and the associated data assimilation components. This new capability is then demonstrated using observations from the Global Precipitation Measurement (GPM) Dual-frequency Precipitation Radar (DPR). The radar forward operator simulates reflectivities that agree well with observations, with mean differences typically below 2 dB during the August 2023–January 2024 test period. Consequently, a large number of DPR observations can be successfully assimilated into GEOS. Case studies of tropical cyclones show that DPR assimilation can substantially improve the analyzed distribution of precipitation and hydrometeors and, in some cases, improve atmospheric state variables such as wind, pressure, temperature, and water vapor. These results demonstrate the potential of spaceborne radar observations to improve the representation of precipitating systems in GEOS and, ultimately, the quality of future reanalyses and weather forecasts.

The launch of the Tropospheric Emissions: Monitoring of Pollution (TEMPO) mission in 2023 marked a new era in air quality monitoring by providing high-frequency, geostationary observations of column NO2 across most of North America. In this study, we present the first implementation of a TEMPO NO2 data assimilation system using the Joint Effort for Data assimilation Integration (JEDI) framework. Leveraging a four-dimensional ensemble variational (4DEnVar) approach and an Ensemble of Data Assimilations (EDA), we demonstrate a novel capability to assimilate hourly NO2 retrievals from TEMPO alongside polar-orbiting TROPOspheric Monitoring Instrument (TROPOMI) data into NASA's GEOS Composition Forecast (GEOS-CF) model. The system is evaluated over the CONUS region for August 2023, using a suite of independent measurements including Pandora spectrometers, AirNow surface stations, and aircraft-based observations from Atmospheric Emissions and Reactions Observed from Megacities to Marine Areas (AEROMMA) and Synergistic TEMPO Air Quality Science (STAQS) field campaigns. Results show that the assimilation system successfully integrates geostationary NO2 observations, improves model performance in the column, and captures diurnal variability. However, assimilation also leads to systematic reductions in NO2 levels, which improves agreement with some data sets (e.g., Pandora, AEROMMA) but degrades comparisons with others (e.g., STAQS). These findings highlight the importance of joint evaluation across platforms and motivate further development of dual-concentration emission assimilation schemes. While the system imposes high computational costs, primarily from the forecast model, ongoing efforts to integrate AI-based model emulators offer a promising path toward scalable, real-time assimilation of geostationary atmospheric composition data.

The Joint Center for Satellite Data Assimilation (JCSDA) is a multiagency research center established to improve the quantitative use of satellite data in atmosphere, ocean, climate and environmental analysis and prediction systems. At the JCSDA, scientists and software engineers within the Joint Effort for Data Assimilation Integration (JEDI) are developing a unified data assimilation framework for research and operational use. To harness the full potential of ever-increasing volumes of data from new and evolving Earth observation systems, a new online database software system has been developed and deployed by the JCSDA. As a core component of JEDI, the Research Repository for Data and Diagnostics (R2D2) performs data registration, management, and configuration services for data assimilation computational workflows. We present an overview of R2D2's distributed system of data stores, SQL data model, intuitive python API, and user support efforts. In addition, we will detail R2D2's utilization by environmental prediction applications developed by the JCSDA and its partners.

An ensemble of 3D ensemble-variational (En-3DEnVar) data assimilations is demonstrated with the Joint Effort for Data assimilation Integration (JEDI) with the Model for Prediction Across Scales – Atmosphere (MPAS-A) (i.e., JEDI-MPAS). Basic software building blocks are reused from previously presented deterministic 3DEnVar functionality and combined with a formal experimental workflow manager in MPAS-Workflow. En-3DEnVar is used to produce an 80-member ensemble of analyses, which are cycled with ensemble forecasts in a 1-month experiment. The ensemble forecasts approximate a purely flow-dependent background error covariance (BEC) at each analysis time. The En-3DEnVar BECs and prior ensemble-mean forecast errors are compared to those produced by a similar experiment that uses the Data Assimilation Research Testbed (DART) ensemble adjustment Kalman filter (EAKF). The experiment using En-3DEnVar produces a similar ensemble spread to and slightly smaller errors than the EAKF. The ensemble forecasts initialized from En-3DEnVar and EAKF analyses are used as BECs in deterministic cycling 3DEnVar experiments, which are compared to a control experiment that uses 20-member MPAS-A forecasts initialized from Global Ensemble Forecast System (GEFS) initial conditions. The experimental ensembles achieve mostly equivalent or better performance than the off-the-shelf ensemble system in this deterministic cycling setting, although there are many obvious differences in configuration between GEFS and the two MPAS ensemble systems. An additional experiment that uses hybrid 3DEnVar, which combines the En-3DEnVar ensemble BEC with a climatological BEC, increases tropospheric forecast quality compared to the corresponding pure 3DEnVar experiment. The JEDI-MPAS En-3DEnVar is technically working and useful for future research studies. Tuning of observation errors and spread is needed to improve performance, and several algorithmic advancements are needed to improve computational efficiency for larger-scale applications.

The Joint Effort for Data assimilation Integration (JEDI), is an innovative data assimilation system for Earth system prediction, spearheaded by the Joint Center for Satellite Data Assimilation (JCSDA) and slated for implementation in major operational modeling systems across the globe in the coming years. Funded as an inter-agency development by NOAA, NASA, the U.S. Navy and Air Force, and with contributions from the UK Met Office, JEDI must operate on a wide range of computing platforms. The recent move towards cloud computing systems puts portability, adaptability and performance across systems, from dedicated High Performance Computing systems to commercial clouds and workstations, in the critical path for the success of JEDI.JEDI is a highly complex application that relies on a large number of third-party software packages to build and run. These packages can include I/O libraries, workflow engines, Python modules for data manipulation and plotting, several ECMWF libraries for complex arithmetics and grid manipulations, and forecast models such as the Unified Forecast System (UFS), the Goddard Earth Observing System (GEOS), the Modular Ocean Model (MOM6), the Model for Prediction across Scales (MPAS), the Navy Environmental Prediction sysTem Utilizing the NUMA corE (NEPTUNE), and the Met Office Unified Model (UM).With more than 100 contributors and rapid code development it is critical to perform thorough automated testing, from basic unit tests to comprehensive end-to-end-tests. This presentation summarizes recent efforts to leverage cloud computing environments for research, development, and near real-time applications of JEDI, as well as for developing a Continuous Integration/Continuous Delivery (CI/CD) pipeline. These efforts rest on a newly developed software stack called spack-stack, a joint effort of JCSDA, the NOAA Environmental Modeling Center (EMC) and the U.S. Earth Prediction Innovation Center (EPIC). Automatic testing in JEDI is implemented with modern software development tools such as GitHub, Docker containers, various Amazon Web Services (AWS), and CodeCov for testing and evaluation of code performance. End-to-end testing is realized in JCSDA’s newly developed Skylab Earth system data assimilation application, which combines JEDI with the Research Repository for Data and Diagnostics (R2D2) and the Experiments and Workflow Orchestration Kit (EWOK), and which leverages the AWS Elastic Compute Cloud (EC2) for testing, research, development and production.

On 24 September 2021, JEDI-MPAS 1.0.0, a new data assimilation (DA) system for the Model Prediction Across Scales – Atmosphere (MPAS-A) built on the software framework of the Joint Effort for Data assimilation Integration (JEDI) was publicly released for community use. Operating directly on the native MPAS unstructured mesh, JEDI-MPAS capabilities include three-dimensional variational (3DVar) and ensemble–variational (EnVar) schemes as well as the ensemble of DA (EDA) technique. On the observation side, one advanced feature in JEDI-MPAS is the full all-sky approach for satellite radiance DA with the introduction of hydrometeor analysis variables. This paper describes the formulation and implementation of EnVar for JEDI-MPAS. JEDI-MPAS 1.0.0 is evaluated with month-long cycling 3DEnVar experiments with a global 30–60 km dual-resolution configuration. The robustness and credible performance of JEDI-MPAS are demonstrated by establishing a benchmark non-radiance DA experiment, then incrementally adding microwave radiances from three sources: Advanced Microwave Sounding Unit-A (AMSU-A) temperature sounding channels in clear-sky scenes, AMSU-A window channels in all-sky scenes, and Microwave Humidity Sounder (MHS) water vapor channels in clear-sky scenes. JEDI-MPAS 3DEnVar behaves well with a substantial and significant positive impact obtained for almost all aspects of forecast verification when progressively adding more microwave radiance data. In particular, the day 5 forecast of the best-performing JEDI-MPAS experiment yields an anomaly correlation coefficient (ACC) of 0.8 for 500 hPa geopotential height, a gap of roughly a half day when compared to cold-start forecasts initialized from operational analyses of the National Centers for Environmental Prediction, whose ACC does not drop to 0.8 until a lead time of 5.5 d. This indicates JEDI-MPAS's great potential for both research and operations.

The Joint Effort for Data assimilation Integration (JEDI) -- led by the Joint Center for Satellite Data Assimilation (JCSDA) -- is an inter-organizational endeavor to develop a common framework for performing data assimilation. This extensive framework will ultimately provide solvers, observation operators, interpolation and model interfaces using object oriented modeling. Two partners involved in JEDI use or plan to use the Finite Volume Cubed-Sphere (FV3) dynamical core to produce weather forecasts; these are NASA's Global Modeling and Assimilation Office and NOAA's National Center for Environment Prediction. In this work we present an update on ongoing efforts to integrate the FV3 tangent linear and adjoint models into the prototype JEDI framework. We setup and run a simple cycled data assimilation experiment using 4DVAR on the cubed sphere grid and with the FV3 tangent linear and adjoint models. Development of the observation operators for JEDI is separately underway. Instead of using real observations a simplified set of simulated observations will be used. We discuss the steps required to bring the FV3 linearized model into the object oriented framework and consider what would be the computational requirements of running this configuration for an operational system. FV3 uses a small time-step to ensure that small scales are well resolved, however this presents design challenges when running 4DVAR with the adjoint. An approach to storing the FV3 model trajectory has been developed that maintains the flexibility of using automatic differentiation. We discuss how this approach is incorporated into the framework. Other important uses of adjoint models include computing observation impacts and singular vectors, we consider how these tools can be included in JEDI.

The numerical solution of saddle point systems has received a lot of attention over the past few years in a wide variety of applications such as constrained optimization, computational fluid dynamics and optimal control, to name a few. In this paper, we focus on the saddle point formulation of a large-scale variational data assimilation problem, where the computations involving the constraint blocks are supposed to be much more expensive than those related to the (1, 1) block of the saddle point matrix. New low-rank limited memory preconditioners exploiting the particular structure of the problem are proposed and analysed theoretically. Numerical experiments performed within the Object-Oriented Prediction System are presented to highlight the relevance of the proposed preconditioners.

The ECMWF twentieth century reanalysis (ERA-20C; 1900-2010) assimilates surface pressure and marine wind observations. The reanalysis is single-member, and the background errors are spatiotemporally varying, derived from an ensemble. The atmospheric general circulation model uses the same configuration as the control member of the ERA-20CM ensemble, forced by observationally based analyses of sea surface temperature, sea ice cover, atmospheric composition changes, and solar forcing. The resulting climate trend estimations resemble ERA-20CM for temperature and the water cycle. The ERA-20C water cycle features stable precipitation minus evaporation global averages and no spurious jumps or trends. The assimilation of observations adds realism on synoptic time scales as compared to ERA-20CM in regions that are sufficiently well observed. Comparing to nighttime ship observations, ERA-20C air temperatures are 1 K colder. Generally, the synoptic quality of the product and the agreement in terms of climate indices with other products improve with the availability of observations. The MJO mean amplitude in ERA-20C is larger than in 20CR version 2c throughout the century, and in agreement with other reanalyses such as JRA-55. A novelty in ERA-20C is the availability of observation feedback information. As shown, this information can help assess the product's quality on selected time scales and regions.

Presently, a preferred minimization for strong-constraint four-dimensional variational (4D-Var) assimilation uses a Lanczos-based conjugate gradient (CG) algorithm. This requires the availability of a square-root of the background-error covariance matrix (B). In the context of weak-constraint 4D-Var, this requirement might be too restrictive for the formulations of the model error term. It might therefore be desirable to avoid a square-root decomposition of the augmented background term. An appealing minimization scheme is the double CG minimization employed, for example, in the grid-point statistical interpolation (GSI) analysis. Realizing the double CG algorithm is a special case of the more general bi-conjugate gradient (BiCG) method for solving non-symmetric problems, the present work introduces a Lanczos-based preconditioning strategy when B, instead of its square-root, is used initially. Implementation of the scheme is done in the context of the GSI analysis system, and preliminary experiments are presented using its 3D-Var version. Comparison of the Lanczos-based CG and the BiCG shows that the algorithms converge at the same rate and to the same solution. Despite the additional computational cost, the importance of the re-orthogonalization step is also shown to be fundamental to any of these CG algorithms. Furthermore, when using the Hessian eigenvectors for preconditioning, the BiCG behaviour is shown to be comparable to that of the Lanczos-CG algorithm. Both schemes construct the same approximation of the Hessian with the same number of eigenvectors, and benefit in the same way from the reduction of the condition number. The efficiency, computational cost, and stability of the three algorithms are discussed. Copyright (c) 2012 Royal Meteorological Society

This presentation discusses an approach to estimate model error using observation residuals. Based on the sequential fixed-lag smoother; we introduce a diagnostic procedure to allow estimating model error over a dense observing system. Optimality considerations are examined in light of the sequential results. The procedure is re-interpreted in the language of variational assimilation, such as 4d-Var. Illustrations of the approach are given by studying both identical-twin and fraternal-twin experimental settings for a system governed by Lorenz-type dynamics. Preliminary results by looking at observation residual statistics for the ECMWF data assimilation system are also shown. The presentation will be part of a series of discussions on issues related to four-dimensional data assimilation under weak-constraint and methodologies to estimate model error.

We describe recent progress in weak-constraint and long-window 4D-Var. Experiments in a two-layer quasi-geostrophic system confirm the results obtained by Fisher et al. (2005) for the Lorenz (1995) model. The experiments show that increasing the length of the analysis window in 4D-Var improves the analysis. Moreover, this benefit is present even if the model error is induced by systematic error in the model parameters, and is imperfectly described by the analysis system. We present results using the full ECMWF analysis system showing that the analysis window can usefully be extended to 24 hours. It is expected that 24 hour 4D-Var will be implemented in 2012. We also present results using a degraded observation network comprising surface pressure observations only. We find that a 24-hour analysis window gives better results than a 12-hour analysis window, and the improvement is even more significant when the 24-hour windows are “overlapped” by performing an analysis every 12 hours. The importance of properly taking into account the mean component of model error is highlighted in both the two-layer quasi-geostrophic context, and in experiments conducted using an Reanalysis configuration of the ECMWF assimilation system in which only historical surface pressure observations are assimilated. A major potential benefit of weak-constraint 4D-Var is that it allows parallelisation of the inner loops of 4D-Var. We discuss how this benefit might be realised, and present a new parallel algorithm which we call the “saddle point” formulation of 4D-Var. We demonstrate the algorithm using the two-layer quasi-geostrophic system. Technical Memorandum No. 655 1 Weak-Constraint and Long-Window 4D-Var

The Global Modeling and Assimilation Office (GMAO) has been working to promote its prototype four-dimensional variational (4DVAR) system to a version that can be exercised at operationally desirable configurations. Beyond a general circulation model (GeM) and an analysis system, traditional 4DV AR requires availability of tangent linear (TL) and adjoint (AD) models of the corresponding GeM. The GMAO prototype 4DVAR uses the finite-volume-based GEOS GeM and the Grid-point Statistical Interpolation (GSI) system for the first two, and TL and AD models derived ITom an early version of the finite-volume hydrodynamics that is scientifically equivalent to the present GEOS nonlinear GeM but computationally rather outdated. Specifically, the TL and AD models hydrodynamics uses a simple (I-dimensional) latitudinal MPI domain decomposition, which has consequent low scalability and prevents the prototype 4DV AR ITom being used in realistic applications. In the near future, GMAO will be upgrading its operational GEOS GCM (and assimilation system) to use a cubed-sphere-based hydrodynamics. This versions of the dynamics scales to thousands of processes and has led to a decision to re-derive the TL and AD models for this more modern dynamics, thus taking advantage of a two-dimensional MPI decomposition and improved scalability properties. With the aid of the Transformation of Algorithms in FORTRAN (l'AF) automatic adjoint generation tool and some hand-coding, a version of the cubed-sphere-based TL and AD models, with a simplified vertical diffusion scheme, is now available, enabling multiple configurations of standard implementations of 4DV AR in GEOS. Concurrent to this development, collaboration with the National Centers for Environmental Prediction (NCEP) and the Earth System Research Laboratory (ESRL) has allowed GMAO to implement a hybrid-ensemble capability within the GEOS data assimilation system. Both 3Dand 4D-ensemble capabilities are presently available thus allowing GMAO to now evaluate the performance and benefit of various ensemble and variational assimilation strategies. This presentation will cover the most recent developments taking place at GMAO and show results from various comparisons from traditional techniques to more recent ensemble-based ones.

Uncertainty in the two main sources of information (observations and model) should in theory be accounted for so that the best data assimilation can do is determine the best estimate of the state of the system given the model, the observations and their error characteristics. In operational practice though, model error is difficult to take into account. It is most of the time hidden in the background error term which, although not strictly an input of the data assimilation system, makes data assimilation practical. We present below an approach to account for model error in 4D-Var and some results in the IFS.

Over the years, data assimilation and forecast schemes have evolved into very complex systems. For example, the ECMWF data assimilation system now handles a ve ry large range of space and surfacebased meteorological observations, combining these with p rior information of the atmospheric state and using a comprehensive linearised forecast model to ensu r that the observations are incorporated in a dynamically consistent way. Meanwhile the ECMWF f orecast model itself represents ever more physical processes, with ever increasing complexity, and is integrated at ever higher resolution. Recognising that any analysis or forecast is actually a probabilistic one, ECMWF continues to incorporate representations of uncertainty into all its fo recast components. Efficient and coordinated diagnosis of such a complex system is a necessity. Diagnostics has always been an active area of research and de velopment at ECMWF. Presently, ECMWF has a small nucleus of scientists dedicated to diagnos tics, who have a broad understanding of the global circulation and an overview of the assimilatio n and forecasting systems. Their unifying role will become ever more important as progress leads, i nevitably, to more fragmentation and specialisation within the overall task of producing foreca sts. However diagnostic work is not only about getting an overview, because all researchers need to p roduce ever more detailed diagnostics within their own particular fields. These trends raise the qu stion of whether we will need to enhance communication, coordination and collaboration across the traditional section boundaries in future. In order to address this issue, and to produce this report, a ‘ Working Group on Diagnostics’ (WGD) has been established at ECMWF. In this report, a diagnostics framework is introduced which highlights key diagnosis areas and their links. From each area, one or two ‘strategic’ diagnostic too ls ( r techniques) are highlighted. These tools are ‘strategic’ in that they are likely to be developed further and will inform future system developments. In order to demonstrate the utility of across -section collaborative work, these tools have been used to collectively address a long-standing prob lem for the ECMWF forecasting systems: that of the over-active Asian monsoon. This case-study prov ides a structural blue-print for future across-section projects focused on other forecasting prob lems and on the assessment of new model cycles. In discussing the above, together with future diagnostic re quir ments, this paper shows where ECMWF stands at present in terms of diagnostic work and proposes a s trategy that will ensure it meets the challenges of the future. Technical Memorandum No. 637 1 Developments in Diagnostics Research