Global and regional hydrological databases, as well as domain-agnostic repositories, play a crucial role in advancing scientific research and decision-making processes. With new and existing data infrastructures such as TERENO and eLTER, as well as governmental monitoring initiatives, efforts to enhance the size, capabilities, and accessibility of these services are underway. However, a key challenge persists across large-scale data collections - the need for rigorous harmonization of diverse data from various sources. This challenge extends beyond obvious considerations like numerical precision and date formats, encompassing more nuanced aspects such as data quality and its representation. Hydrological time series data, often acquired from remote sensors and monitoring stations, are susceptible to errors arising from sensor malfunctions, anomalies, and environmental fluctuations. Unchecked, these inaccuracies can lead to erroneous results and compromise decision-making processes. Addressing this critical issue, the System for Automated Quality Control - SaQC emerges as a pioneering solution, offering a comprehensive tool/framework for automated and customizable quality control and processing of time series data. SaQC empowers researchers and practitioners in the hydrological sciences, providing a convenient and efficient means to identify and rectify data anomalies. In addition to a large body of built-in routines and algorithms, the framework's extensibility allows users to implement custom quality check routines and schemes, tailoring the quality control process to specific research objectives and the evolving needs of data services. This presentation delves into the core principles of SaQC, showcasing its flexibility in handling diverse data types and adapting to various hydrological monitoring scenarios. Through real-world examples of fully automatized quality control and data processing workflows, we highlight the benefits of SaQC in enhancing data integrity, reducing manual intervention, and expediting the analysis pipeline. SaQC not only identifies anomalies but also provides a systematic and transparent approach to data quality assurance, contributing to the overall reliability of hydrological datasets. Lennart Schmidt, David Schäfer, Juliane Geller, Peter Lünenschloss, Bert Palm, Karsten Rinke, Corinna Rebmann, Michael Rode, Jan Bumberger, System for automated Quality Control (SaQC) to enable traceable and reproducible data streams in environmental science, Environmental Modelling & Software, Volume 169, 2023, 105809, ISSN 1364-8152, https://doi.org/10.1016/j.envsoft.2023.105809.

Efficient quality control (QC) of time series data from environmental sensors is crucial for ensuring data accuracy and reliability. In this work, we turn to machine learning, specifically Graph Neural Networks (GNN), to elevate QC efficiency for large datasets originating from sparsely distributed sensors. Our proposed model, specifically tailored for anomaly detection as a vital aspect of QC, combines graph convolution (GC) and Long Short-Term Memory (LSTM) layers to capture both spatial dependencies and temporal patterns in the time series data. The focus on anomaly detection enables the identification of deviations or irregularities in the signal, providing insights into important events, faults, or disturbances within the data.We conducted experiments using two distinct types of labeled data: three months of data in 2019 from 20 Commercial Microwave Links (CML) distributed around Germany and a 2.5-year period (June 2014 to December 2016) of soil moisture data from the TERENO SoilNet network in Hohes Holz, Germany. These datasets, encompassing an impressive 2.5 million samples, pose challenges in QC due to diverse dynamics, signal anomalies, and variations in temporal resolution and spatial densities of observations. The classification results demonstrated satisfactory performance, with Matthews Correlation Coefficients of over 0.6 and 0.8 for the CML and SoilNet datasets, respectively. To evaluate the added value of processing the spatial information provided by neighboring sensors, we also compared the results of our final GNN with a baseline model that uses the same LSTM layers but disregards the GC layer, which integrates the neighboring information. The GNN model exhibited improved performance, as evidenced by 5-fold cross-validation mean Area Under the Receiver Operating Characteristic Curve (AUC) values of 0.934 and 0.971 for the CML and SoilNet data, respectively. In contrast, the baseline model yielded mean AUC values of 0.877 and 0.950, highlighting the effectiveness of incorporating the information from neighboring sensors via the GC layers to enhance anomaly detection for environmental sensor time series data.

Environmental sensor networks play a crucial role in monitoring key parameters essential for understanding Earth’s systems. To ensure the reliability and accuracy of collected data, effective quality control (QC) measures are essential. Conventional QC methods struggle to handle the complexity of environmental data. Conversely, advanced techniques such as neural networks, are typically not designed to process data from sensor networks with irregular spatial distribution. In this study, we focus on anomaly detection in environmental sensor networks using graph neural networks, which can represent sensor network structures as graphs. We investigate its performance on two datasets with distinct dynamics and resolution: commercial microwave link (CML) signal levels used for rainfall estimation and SoilNet soil moisture measurements. To evaluate the benefits of incorporating neighboring sensor information for anomaly detection, we compare two models: Graph Convolution Network (GCN) and a graph-less baseline-long short- term memory (LSTM). Our robust evaluation through 5-fold cross-validation demonstrates the superiority of the GCN models. For CML, the mean area under curve values for the GCN was 0.941 compared to 0.885 for the baseline-LSTM, and for SoilNet, it was 0.858 for GCN and 0.816 for the baseline-LSTM. Visual inspection of CML time series revealed that the GCN proficiently classified anomalies and remained resilient against rain-induced events often misidentified by the baseline- LSTM. However, for SoilNet, the advantage of GCN was less pronounced likely due to a fragile labeling strategy. Through interpretable model analysis, we demonstrate how feature attributions vividly illustrate the significance of neighboring sensor data, particularly in distinguishing between anomalies and expected changes in signal level in the time series.

The rapidly growing number of low-cost environmental sensors and data from opportunistic sensors constantly advances the quality as well as the spatial and temporal resolution of weather and climate models. However, it also leads to the need for effective tools to ensure the quality of collected data.Time series quality control (QC) from multiple spatial, irregularly distributed sensors is a challenging task, as it requires the simultaneous integration and analysis of observations from sparse neighboring sensors and consecutive time steps. Manual QC is very often time- and labour- expensive and requires expert knowledge, which introduces subjectivity and limits reproducibility. Therefore, automatic, accurate, and robust QC solutions are in high demand, where among them one can distinguish machine learning techniques. In this study, we present a novel approach for the quality control of time series data from multiple spatial, irregularly distributed sensors using graph neural networks (GNNs). Although we applied our method to commercial microwave link attenuation data collected from a network in Germany between April and October 2021, our solution aims to be generic with respect to the number and type of sensors, The proposed approach involves the use of an autoencoder architecture, where the GNN is used to model the spatial relationships between the sensors, allowing for the incorporation of contextual information in the quality control process. While our model shows promising results in initial tests, further research is needed to fully evaluate its effectiveness and to demonstrate its potential in a wider range of environmental applications. Eventually, our solution will allow us to further foster the observational basis of our understanding of the natural environment.

Environmental sensor networks produce ever-growing volumes of time series data with great potential to broaden the understanding of complex spatiotemporal environmental processes. However, this growth also imposes its own set of new challenges. Especially the error-prone nature of sensor data acquisition is likely to introduce disturbances and anomalies into the actual environmental signal. Most applications of such data, whether it is used in data analysis, as input to numerical models or modern data science approaches, usually rely on data that complies with some definition of quality.To move towards high-standard data products, a thorough assessment of a dataset's quality, i.e., its quality control, is of crucial importance. A common approach when working with time series data is the annotation of single observations with a quality label to transport information like its reliability. Downstream users and applications are hence able to make informed decisions, whether a dataset in its whole or at least parts of it are appropriatefor the intended use.Unfortunately, quality control of time series data is a non-trivial, time-consuming, scientifically undervalued endeavor and is often neglected or executed with insufficient rigor. The presented software, the System for automated Quality Control (SaQC), provides all basic and many advanced building blocks to bridge the gap between data that is usually faulty but expected to be correct in an accessible, consistent, objective and reproducible way. Its user interfaces address different audiences ranging from the scientific practitioner with little access to the possibilities of modern software development to the trained programmer. SaQC delivers a growing set of generic algorithms to detect a multitude of anomalies and to process data using resampling, aggregation, and data modeling techniques. However, one defining component of SaQC is its innovative approach to storing runtime process information. In combination with a flexible quality annotation mechanism, SaQC allows to extend quality labels with fine-grained provenance information appropriate to fully reproduce the system's output.SaQC is proving its usefulness on a daily basis in a range of fully automated data flows for large environmental observatories. We highlight use cases from the TERENO Network, showcasing how reproducible automated quality control can be implemented into real-world, large-scale data processing workflows to provide environmental sensor data in near real-time to data users, stakeholders and decision-makers.

Environmental sensor networks produce continuously increasing volumes of raw data that need to be transformed into usable data for monitoring ongoing environmental changes and decision-support. The crucial challenge is providing data in real-time, which requires the rigorous automation of quality control (QC) workflows using suitable software tools. We present the System for automated Quality Control (SaQC), a software package for the automated quality control of environmental time series data that is universal and that can be expanded in its set of domain-agnostic QC and processing functionalities, while at the same time being user-friendly in its low-code configuration environment. Two applications present the configuration of basic and advanced quality control applications using SaQC. We also elaborate on the explicit user controls over the handling of quality flags and how SaQC can be used to make QC-workflows traceable and reproducible, thus promoting FAIR data streams of high quality.

For the reduction of climate change and the understanding of the effects of anthropogenic interventions on environmental systems, the monitoring of these systems is a fundamental requirement that relies heavily on the availability of extensive but consistent data sets.Quality control tests and consistency routines that generate those datasets from available sensor data will inevitably produce data gaps, where measurement data does not pass tests or is simply not available.However, most further data utilization will need those data gaps to be filled (imputed) in a consistent way. This consistency usually is assured by having a good set of predictors, together with a suitable method for predicting the variable that is to be imputed.This is also true for precipitation, a meteorologic variable that is fundamental to the understanding of hydro logical system dynamics but notoriously hard to predict at the micro climatic scale, with sampling rates exceeding the one hour mark.We conducted an imputation study with machine learning methods on precipitation time series collected in a reference set of gauging stations that are a subset of the wider network of the german meteorologic service (DWD), where precipitation and other meteorological data is available at a 10 minute sampling rate.We trained an Extreme Gradient Boosted Tree classifier and a Deep Neural Network regressor on a 10 years record of those data. We selected several distinct sets of predictors available in the surrounding of the reference station based on temporal and spatial proximity and evaluated the feature importance at different proximity value levels.Assuming that the imputation does not have to be performed at real time, but serves as a post-processing step, we could extend the set of bounding conditions to measurements obtained in the future of the gap to be imputed, and could thus improve over results obtained in regular forecasting scenarios.To further improve the imputation results, especially for the matching of singular and erratic rainfall events, we aligned spatio-temporally separated measurements of the same (traveling) rainfall events by including a non-linear time series stretching algorithm (dynamic time warping) into the samples preprocessing.We observed, that meteorologic variables such as wind and humidity, that are useful for the prediction of precipitation at lower sampling rates, can not compensate for the noise their inclusion in the set of predictors results in, when imputing precipitation sampled at a 10 minutes rate.However, with precipitation collected at neighboring stations used as predictors and the preprocessing measures taken, we were able to achieve a solid correlation score and could thus show, that ML-driven post processing routines enable imputations at high temporal resolutions, providing the end user with consistent precipitation data sets.

The number of sensors used in the environmental system sciences is increasing rapidly, and while this trend undoubtedly provides a great potential to broaden the understanding of complex spatio-temporal processes, it comes with its own set of new challenges. The flow of data from a source to its sink, from sensors to databases, involves many, usually error prone intermediate steps. From the data acquisition with its specific scientific and technical challenges, over the data transfer from often remote locations to the final data processing, all carry great potential to introduce errors and disturbances into the actual environmental signal.Quantifying these errors becomes a crucial part of the later evaluation of all measured data. While many large environmental observatories are moving from manual to more automated ways of data processing and quality assurance, these systems are usually highly customized and hand written. This approach is non-ideal in several ways: First, it is a waste of resources as the same algorithms are implemented over and over again and second, it imposes great challenges to reproducibility. If the relevant programs are made available at all, they expose all problems of software reuse: correctness of the implementation, readability and comprehensibility for future users, as well as transferability between different computing environments. Beside these problems, related to software development in general, another crucial factor comes into play: the end product, a processed and quality controlled data set, is closely tied to the current version of the programs in use. Even small changes to the source code can lead to vastly differing results. If this is not approached responsibly, data and programs will inevitably fall out of sync.The presented software, the 'System for automated Quality Control (SaQC)' (www.ufz.git.de/rdm-software/saqc), helps to either solve, or massively simplify the solution to the presented challenges. As a mainly no-code platform with a large set of implemented functionality, SaQC lowers the entry barrier for the non-programming scientific practitioner, without sacrificing the possibilities to fine-grained adaptation to project specific needs. The text based configuration allows the easy integration into version control systems and thus opens the opportunity to use well established software for data lineage. We will give a short overview of the program's unique features and showcase possibilities to build reliable and reproducible processing and quality assurance pipelines for real-world data from a spatially distributed, heterogeneous sensor network.

With more and more data being gathered from environmental sensor networks, the importance of automated quality-control (QC) routines to provide usable data in near-real time is becoming increasingly apparent. Machine-learning (ML) algorithms exhibit a high potential to this respect as they are able to exploit the spatio-temporal relation of multiple sensors to identify anomalies while allowing for non-linear functional relations in the data. In this study, we evaluate the potential of ML for automated QC on two spatio-temporal datasets at different spatial scales: One is a dataset of atmospheric variables at 53 stations across Northern Germany. The second dataset contains timeseries of soil moisture and temperature at 40 sensors at a small-scale measurement plot.Furthermore, we investigate strategies to tackle three challenges that are commonly present when applying ML for QC: 1) As sensors might drop out, the ML models have to be designed to be robust against missing values in the input data. We address this by comparing different data imputation methods, coupled with a binary representation of whether a value is missing or not. 2) Quality flags that mark erroneous data points to serve as ground truth for model training might not be available. And 3) There is no guarantee that the system under study is stationary, which might render the outputs of a trained model useless in the future. To address 2) and 3), we frame the problem both as a supervised and unsupervised learning problem. Here, the use of unsupervised ML-models can be beneficial as they do not require ground truth data and can thus be retrained more easily should the system be subject to significant changes. In this presentation, we discuss the performance, advantages and drawbacks of the proposed strategies to tackle the aforementioned challenges. Thus, we provide a starting point for researchers in the largely untouched field of ML application for automated quality control of environmental sensor data.