A novel ground-based radiometer, referred to as the Shortwave Array Spectroradiometer-Hemispheric (SAS-He), is introduced. This radiometer uses the shadow-band technique to report total irradiance and its direct and diffuse components frequently (every 30 s) with continuous spectral coverage (350-1700 nm) and moderate spectral (similar to 2.5 nm ultraviolet-visible and similar to 6 nm shortwave-infrared) resolution. The SAS-He's performance is evaluated using integrated datasets collected over coastal regions during three field campaigns supported by the US Department of Energy's Atmospheric Radiation Measurement (ARM) program, namely the (1) Two-Column Aerosol Project (TCAP; Cape Cod, Massachusetts), (2) Tracking Aerosol Convection Interactions Experiment (TRACER; in and around Houston, Texas), and (3) Eastern Pacific Cloud Aerosol Precipitation Experiment (EPCAPE; La Jolla, California). We compare (i) aerosol optical depth (AOD) and total optical depth (TOD) derived from the direct irradiance, as well as (ii) the diffuse irradiance and direct-to-diffuse ratio (DDR) calculated from two components of the total irradiance. As part of the evaluation, both AOD and TOD derived from the SAS-He direct irradiance are compared to those provided by a collocated Cimel sunphotometer (CSPHOT) at five (380, 440, 500, 675, 870 nm) and two (1020, 1640 nm) wavelengths, respectively. Additionally, the SAS-He diffuse irradiance and DDR are contrasted with their counterparts offered by a collocated multifilter rotating shadowband radiometer (MFRSR) at six (415, 500, 615, 675, 870, 1625 nm) wavelengths. Overall, reasonable agreement is demonstrated between the compared products despite the challenging observational conditions associated with varying aerosol loadings and diverse types of aerosols and clouds. For example, the AOD- and TOD-related values of root mean square error remain within 0.021 at 380, 440, 500, 675, 870, 1020, and 1640 nm wavelengths during the three field campaigns.

Airborne measurements are pivotal for providing detailed, spatiotemporally resolved information about atmospheric parameters and aerosol and cloud properties, thereby enhancing our understanding of dynamic atmospheric processes. For 30 years, the US Department of Energy (DOE) Office of Science supported an instrumented Gulfstream 1 (G-1) aircraft for atmospheric field campaigns. Data from the final decade of G-1 operations were archived by the Atmospheric Radiation Measurement (ARM) Data Center and made publicly available at no cost to all registered users. To ensure a consistent data format and to improve the accessibility of the ARM airborne data, an integrated dataset was recently developed covering the final 6 years of G-1 operations (2013 to 2018, 10.5439/1999133; Mei and Gaustad, 2024). The integrated dataset includes data collected from 236 flights (766.4 h), which covered the Arctic, the US Southern Great Plains (SGP), the US West Coast, the eastern North Atlantic (ENA), the Amazon Basin in Brazil, and the Sierras de Cordoba range in Argentina. These comprehensive data streams provide much-needed insight into spatiotemporal variability in the thermodynamic quantities and aerosol and cloud properties for addressing essential science questions in Earth system process studies. This paper describes the DOE ARM merged G-1 datasets, including information on the acquisition, data collection challenges and future potentials, and quality control processes. It further illustrates the usage of this merged dataset to evaluate the Energy Exascale Earth System Model (E3SM) with the Earth System Model Aerosol-Cloud Diagnostics (ESMAC Diags) package.

Aerosols, tiny suspended liquid and solid particles in air, are involved in several critical processes in the atmosphere. They directly affect the thermodynamic profile of the atmosphere through scattering and absorption of radiation, and indirectly by modulating cloud formation and life cycle. The U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) Climate Research Facility has deployed Aerosol Observation Systems (AOS) containing numerous instruments to measure aerosol optical properties and related parameters at multiple sites worldwide. The ARM Aerosol Observation Systems include measurements of aerosol extensive properties (AEP). Extensive properties are bulk properties that dependent on size, mass, or amount. Also of interest are intensive properties that are inherent, normalized, and independent of amount or bulk quantity. AEPs include scattering and absorption coefficients. Aerosol intensive properties (AIP) include single-scattering albedo, back scatter fraction, asymmetry parameter, and angstrom exponents representing the wavelength dependence of optical properties. The Aerosol Optical Properties (AOP) value-added product (VAP) applies final calibrations and corrections to measurements from the AOS to yield final AEP and related AIP. Since the instruments from these AOS systems operate and deliver data at different flow and operational conditions, extensive checks and processing via software are required so the resulting datastreams are comparable. This document describes the data flow and processing of the AOP VAP applied to aosaop1flynn VAP from either suite to yield a uniform data product with consistent application of calibrations and corrections.

Aerosol optical depth (AOD) derived from hyperspectral measurements can serve as an invaluable input for simultaneous retrievals of particle size distributions and major trace gases. The required hyperspectral measurements are provided by a new ground-based radiometer, the so-called Shortwave Array Spectroradiometer-Hemispheric (SAS-He), recently developed with support from the Department of Energy (DOE) Office Atmospheric Radiation Measurement (ARM) Program. The SAS-He has wide spectral coverage (350-1700nm) and high spectral resolution: about 2.4 nm and 6 nm within 350-1000 nm and 970-1700 nm spectral ranges, respectively. To illustrate an initial performance of the SAS-He, we take advantage of integrated dataset collected during the ARM-supported Two-Column Aerosol Project (TCAP) over the US coastal region (Cape Cod, Massachusetts). This dataset includes AODs derived using data from Aerosol Robotic Network (AERONET) sunphotometer and Multi-Filter Rotating Shadowband Radiometer (MFRSR). We demonstrate that, on average, the SAS-He AODs closely match the MFRSR and AERONET AODs in the ultraviolet and visible spectral ranges for this area with highly variable AOD. Also, we discuss corrections of SAS-He total optical depth for gas absorption in the near-infrared spectral range and their operational implementation.

The Atmospheric Radiation Measurement (ARM) Data Integrator (ADI) is a framework designed to streamline the development of scientific algorithms that analyze, and models that use time-series NetCDF data. ADI automates the process of retrieving and preparing data for analysis, provides a modular, flexible framework that simplifies software development, and supports a data integration workflow. Algorithm and model input data, preprocessing, and output data specifications are defined through a graphical interface. ADI includes a library of software modules to support the workflow, and a source code generator that produces C, IDL®, and Python™ templates to jump start development. While developed for processing climate data, ADI can be applied to any time-series data. This paper discusses the ADI framework, and how ADI's capabilities can decrease the time and cost of implementing scientific algorithms allowing modelers and scientists to focus their efforts on their research rather than preparing and packaging data.

The Atmospheric Radiation Measurement (ARM) Climate Research Facility performs routine in situ and remote-sensing observations to provide a detailed and accurate description of the Earth atmosphere in diverse climate regimes. The result is a diverse data sets containing observational and derived data, currently accumulating at a rate of 30 TB of data and 150,000 different files per month (http://www.archive.arm.gov/stats/storage2.html). Continuing the current processing while scaling this to even larger sizes is extremely important to the ARM Facility and requires consistent metadata and data standards. The standards described in this document will enable development of automated analysis and discovery tools for the ever-growing volumes of data. It also will enable consistent analysis of the multiyear data, allow for development of automated monitoring and data health status tools, and facilitate development of future capabilities for delivering data on demand that can be tailored explicitly to user needs. This analysis ability will only be possible if the data follows a minimum set of standards. This document proposes a hierarchy that includes required and recommended standards.

This paper describes a standard for the representation of data quality information relevant to a geophysical data set, and the specific implementation of this standard for the U.S. Department of Energy's Atmospheric Radiation Measurement (ARM) program. The standard is applicable to near real-time instrument data as well as episodic or batch-processed model data. It was designed especially to facilitate use of data quality checks or "QC" by automated processes but also simplifies interpretation and visualization of this data quality information for humans. The standard is extensible and customizable, permitting an arbitrary number of data quality checks, either global in nature or specific to individual data variables. For the ARM data set, this standard has been implemented in netCDF, which permits arbitrarily verbose descriptions of the data quality checks, while storing the results in a compact, efficient format. In addition, we describe tools developed for the creation and population of the QC fields, rule-based validation of adherence to the QC standards, and the display of QC information in ARM data sets.

Global climate researchers rely upon many forms of sensor data and analytical methods to help profile subtle changes in climate conditions. The U.S. Department of Energy’s Atmospheric Radiation Measurement (ARM) program provides researchers with a collection of curated Value Added Products (VAPs) resulting from continuous sensor data streams, data fusion, and modeling. We are leveraging the Open Provenance Model as a foundational construct that serves the needs of both the VAP producers and consumers. We are organizing the provenance in different tiers of granularity to model VAP lineage, causality at the component level within a VAP, and the causality for each time step as samples are being assembled within the VAP. This paper shares our implementation strategy and how the ARM operations staff and the climate research community can greatly benefit from this approach to more effectively assess and quantify VAP provenance.