Floods are an increasingly common global threat, causing emergencies and severe damage to infrastructure. During crises, organisations such as the World Food Programme use remotely sensed imagery, typically obtained through drones, for rapid situational analysis to plan life-saving actions. Computer Vision tools are needed to support task force experts on-site in the evaluation of the imagery to improve their efficiency and to allocate resources strategically. We introduce the BlessemFlood21 dataset to stimulate research on efficient flood detection tools. The imagery was acquired during the 2021 Erftstadt-Blessem flooding event and consists of high-resolution and georeferenced RGB-NIR images. In the resulting RGB dataset, the images are supplemented with detailed water masks, obtained via a semi-supervised human-in-the-loop technique, where in particular the NIR information is leveraged to classify pixels as either water or non-water. We evaluate our dataset by training and testing established Deep Learning models for semantic segmentation. With BlessemFlood21 we provide labeled high-resolution RGB data and a baseline for further development of algorithmic solutions tailored to flood detection in RGB imagery.

Flooding is one of the most common natural disasters, causing emergencies and significant damage to infrastructure. Amid crises, entities such as the World Food Program employ remotely sensed images, usually via drones, to analyze the situation quickly and plan life-saving interventions. Suitable Computer Vision (CV) tools are necessary to assist experts of first responder teams with the analysis of the image data, increasing their productivity and enabling strategic resource allocation. Many state-of-the-art CV-tools use supervised Deep Learning (DL) techniques, for which labeled training data is needed. In this paper, we present the BlessemFlood21 dataset to support the development of DL-based CV-tools for the task of advanced flood analysis in non-coastal river flood scenarios. In particular, we address the problem of segmenting water from RGB images. The presented resulting dataset consists of high-definition, georeferenced RGB images labeled with precise water masks. The water masks are derived using a proposed semi-automatic human-in-the-loop strategy based on the use of additionally imaged NIR data together with a classical Random Forest approach. We assess the resulting dataset and provide a baseline for future advancements in flood mapping by training and evaluating three different state-of-the-art segmentation models (UNet++, DeepLabV3+, SegFormer-B5). Further, we showcase the potential use of the dataset with several experiments. In particular, we compare with the Floodnet dataset and, we employ the segmentation models trained on our dataset together with a Digital Elevation Model to showcase the potential to estimate flood water levels.

The accurate and comprehensive mapping of land cover has become a central task in modern environmental research, with increasing emphasis on machine learning approaches. However, a clear technical definition of the land cover class is a prerequisite for learning and applying a machine learning model. One of the challenging classes is naturalness and human influence, yet mapping it is important due to its critical role in biodiversity conservation, habitat assessment, and climate change monitoring. We present an interpretable machine learning approach to map patterns related to territorial protected and anthropogenic areas as proxies of naturalness and human influence using satellite imagery. To achieve this, we train a weakly-supervised convolutional neural network and subsequently apply attribution methods such as Grad-CAM and occlusion sensitivity mapping. We propose a novel network architecture that consists of an image-to-image network and a shallow, task-specific head. Both sub-networks are connected by an intermediate layer that captures high-level features in full resolution, allowing for detailed analysis with a wide range of attribution methods. We further analyze how intermediate layer activations relate to their attributions across the training dataset to establish a consistent relationship. This makes attributions consistent across different scenes and allows for a large-scale analysis of remote sensing data. The results highlight that our approach is a promising way to observe and assess naturalness and territorial protection.

Non-invasive investigation of surfaces from drones and manned aircrafts used as camera platforms is a well-established remote-sensing practice. However, cross-comparison of multispectral reflectance from different camera systems across different platforms, locations, and times can be challenging. We investigate reflectance retrieved from Sentinel-2 and two airborne camera systems with respect to the mobile, radiometrically calibrated, two-channel hemispherical-conical field-spectrometer system RoX. This spectrometer system serves in combination with a nine-panel grey scale as ground reference and transfer instrument. In the first step, the ground reference was validated against Sentinel-2 reflectance including atmospheric compensation. Our results suggest significant differences in the uncorrected reflectance from the two airborne sensors with respect to instantaneous calibration across 22 mixed targets. In the second step, those differences were reduced to a median discrepancy below 10% using the proposed in-field empirical line correction method (ELC). Continuous irradiance correction further improved the agreement across the validation targets and achieved a coherent reflectance dataset from all four different sensor systems, from the satellite level to the ground and airborne level, considering the limitations of instrument and in-field handling. NDVI maps created from drone and manned aircraft achieved an agreement around 89% and 95% compared to the satellite after calibration and correction. We consider in-field calibration with additional, continuous down-welling radiance correction of reflectance promising to support fusion of information across four sensors and platforms. Thus, field-spectrometer systems serve as transfer instruments and bridge the gap of information from the satellite down to the ground and airborne scale in future airborne mapping and classification efforts.

The need for accurate yield estimates for viticulture is becoming more important due to increasing competition in the wine market worldwide. One of the most promising methods to estimate the harvest is berry counting, as it can be approached non-destructively, and its process can be automated. In this article, we present a method that addresses the challenge of occluded berries with leaves to obtain a more accurate estimate of the number of berries that will enable a better estimate of the harvest. We use generative adversarial networks, a deep learning-based approach that generates a highly probable scenario behind the leaves exploiting learned patterns from images with non-occluded berries. Our experiments show that the estimate of the number of berries after applying our method is closer to the manually counted reference. In contrast to applying a factor to the berry count, our approach better adapts to local conditions by directly involving the appearance of the visible berries. Furthermore, we show that our approach can identify which areas in the image should be changed by adding new berries without explicitly requiring information about hidden areas.

Data imputation of incomplete image sequences is an essential prerequisite for analyzing and monitoring all development stages of plants in precision agriculture. For this purpose, we propose a conditional Wasserstein generative adversarial network TransGrow that combines convolutions for spatial modeling and a transformer for temporal modeling, enabling time-dependent image generation of above-ground plant phenotypes. Thereby, we achieve the following advantages over comparable data imputation approaches: (1) The model is conditioned by an incomplete image sequence of arbitrary length, the input time points, and the requested output time point, allowing multiple growth stages to be generated in a targeted manner; (2) By considering a stochastic component and generating a distribution for each point in time, the uncertainty in plant growth is considered and can be visualized; (3) Besides interpolation, also test-extrapolation can be performed to generate future plant growth stages. Experiments based on two datasets of different complexity levels are presented: Laboratory single plant sequences with Arabidopsis thaliana and agricultural drone image sequences showing crop mixtures. When comparing TransGrow to interpolation in image space, variational, and adversarial autoencoder, it demonstrates significant improvements in image quality, measured by multi-scale structural similarity, peak signal-to-noise ratio, and Fréchet inception distance. To our knowledge, TransGrow is the first approach for time- and image-dependent, high-quality generation of plant images based on incomplete sequences.

Wilderness areas offer important ecological and social benefits and there are urgent reasons to discover where their positive characteristics and ecological functions are present and able to flourish. We apply a novel explainable machine learning technique to satellite images which show wild and anthropogenic areas in Fennoscandia. Occluding certain activations in an interpretable artificial neural network we complete a comprehensive sensitivity analysis regarding wild and anthropogenic characteristics. This enables us to predict detailed and high-resolution sensitivity maps highlighting these characteristics. Our artificial neural network provides an interpretable activation space increasing confidence in our method. Within the activation space, regions are semantically arranged. Our approach advances explainable machine learning for remote sensing, offers opportunities for comprehensive analyses of existing wilderness, and has practical relevance for conservation efforts.

Object detection is a core task for image analysis and inter-pretation and is broadly applied in applications relying on space- and airborne imagery. Like all supervised deep learning methods, training an object detector generally requires a large amount of representative annotated data, which can be hard to acquire in practice. To overcome this challenge, generating synthetic data can be an option to alleviate a lack of real-world annotated data. One key influential factor for the quality of the synthetic data is the background. We show that the detectors' classifier especially depends severely on the background and has a large impact on the detection preci-sion. Using real background is a natural option, however, we show that this naive approach has drawbacks such as a sig-nificant drop in recall. In this paper, we demonstrate that by using style transfer to match the synthetic foreground to the real background, the detector can mitigate these drawbacks and achieve a more balanced result in terms of precision and recall.

Airborne remote sensing with optical sensor systems is an essential tool for a variety of environmental monitoring applications. Depending on the size of the area to be monitored, either unmanned (UAVs) or manned aircraft are more suitable. For survey areas starting at several square kilometers, piloted aircraft remain the preferred carrier platform. However, a specific class of manned aircraft is often not considered: the gyrocopter-type ultralight aircraft. These aircraft are less expensive to operate than conventional fixed wings. Additionally, they are highly maneuverable, offer a high payload and a long endurance, and thus perfectly fill the niche between UAVs and conventional aircraft. Therefore, the authors have developed a modular and easy-to-use sensor carrier system, the FlugKit, to temporarily convert an AutoGyro MTOsport gyrocopter into a full-fledged aerial remote sensing platform mainly for vegetation monitoring. Accordingly, various suitable optical sensor systems in the visible (VIS), near-infrared (NIR), and longwave infrared (LWIR) were explicitly developed for this carrier system. This report provides a deeper insight into the individual components of this remote sensing solution based on a gyrocopter as well as application scenarios already carried out with the system.

Object detection in aerial images is an important task in environmental, economic, and infrastructure-related tasks. One of the most prominent applications is the detection of vehicles, for which deep learning approaches are increasingly used. A major challenge in such approaches is the limited amount of data that arises, for example, when more specialized and rarer vehicles such as agricultural machinery or construction vehicles are to be detected. This lack of data contrasts with the enormous data hunger of deep learning methods in general and object recognition in particular. In this article, we address this issue in the context of the detection of road vehicles in aerial images. To overcome the lack of annotated data, we propose a generative approach that generates top-down images by overlaying artificial vehicles created from 2D CAD drawings on artificial or real backgrounds. Our experiments with a modified RetinaNet object detection network show that adding these images to small real-world datasets significantly improves detection performance. In cases of very limited or even no real-world images, we observe an improvement in average precision of up to 0.70 points. We address the remaining performance gap to real-world datasets by analyzing the effect of the image composition of background and objects and give insights into the importance of background.

In order to enable the development of powerful machine learning methods for remote sensing-based Earth observation tasks, benchmarks are needed to evaluate the methods and compare them to other methods comprehensively. We present ArtifiVe-Potsdam, a freely available dataset that is targeting vehicle detection in aerial imagery. In particular, the benchmark focuses on enriching real datasets with artificial data and quantifying the added value. The dataset aims to stimulate research on the efficient and cost-effective creation and enrichment of datasets for remote sensing since datasets with a limited number of labels are common and the collection of data and labels is time-consuming and expensive.

Explainable machine learning has recently gained attention due to its contribution to understanding how a model works and why certain decisions are made. A so far less targeted goal, especially in remote sensing, is the derivation of new knowledge and scientific insights from observational data. In our paper, we propose an explainable machine learning approach to address the challenge that certain land cover classes such as wilderness are not well-defined in satellite imagery and can only be used with vague labels for mapping. Our approach consists of a combined U-Net and ResNet-18 that can perform scene classification while providing at the same time interpretable information with which we can derive new insights about classes. We show that our methodology allows us to deepen our understanding of what makes nature wild by automatically identifying simple concepts such as wasteland that semantically describes wilderness. It further quantifies a class’s sensitivity with respect to a concept and uses it as an indicator for how well a concept describes the class.

Over the Hahnöfer Nebenelbe, a part of the Elbe estuary near Hamburg, Germany, a combined aerial survey with an unmanned aerial system (UAV) and a gyrocopter was conducted to acquire information about the water surface temperatures. The water temperature in the estuary is important for biological processes and living conditions of riverine organisms. This study aimed to develop a workflow that allows for comparing and analysing surface temperatures acquired by two different remote sensing systems. The thermal infrared (TIR) datasets were compared with in situ measurements gathered during the data acquisition, where both TIR datasets showed a varying bias. Potential error sources regarding the absolute and relative accuracy were investigated and modelled based on the available measurements, including emissivity, atmosphere, skin effect at the water surface, camera flat field correction and calibration. The largest effects on the observed TIR water temperature had the camera calibration and the modelled atmospheric effects. After the correction steps, both datasets could be combined to create a multitemporal representation of the temperature pattern and profiles over the survey area’s wadden flats.

Up-to-date information about vegetation types and hydromorphological structures and features are essential for the management of waterways. They are e.g. used for the monitoring and reporting of riparian statuses and their changes e.g. after river restoration and consequently, numerous man-days are spent on field surveys. To allow for an effective survey of vegetation and hydromorphology in large or even inaccessible areas, a data acquisition and processing workflow is being developed complementing in-situ methods with remote sensing techniques. This is part of the joint research project “mDRONES4rivers” funded by the German Federal Ministry of Transport and Digital Infrastructure (19F2054A). Aerial surveys by unmanned aerial systems (UAS) and a gyrocopter are combined with ground measurements of hyperspectral reflectance signatures as well as with field mapping of vegetation types and hydromorphological structures and features. The remote sensing data is classified with an object based image analysis and classification algorithm. The mobile and (at selected sites) permanent measurements of hyperspectral field data and the typical field surveys provide data for calibration. Contrary to other approaches that focus on what can be detected and classified with certain sensor systems and datasets, the project addresses equally the user needs to obtain certain classes for monitoring and reporting. The intended results are (i) data acquisition, correction and classification workflow combining remote sensing and field data, identification and change detection (ii) of important vegetation and biotope types and (iii) of hydromorphological structures and substrate as well as indicators necessary for the evaluation of the hydromorphological quality. The preliminary results to be presented include datasets from UAS, gyrocopter, and field surveys, an outline of processing workflow and classification algorithm based on Python scripts and eCognition software and first vegetation and hydromorphological classification results from spring and summer datasets. In conclusion, procedures and algorithms are developed to use remote sensing in combination with and for the reduction of time-consuming traditional field surveys as a future operational tool for monitoring riparian vegetation and structures.

Detecting objects in aerial images is an important task in different environmental and infrastructure-related applications. Deep learning object detectors like RetinaNet offer decent detection performance; however, they require a large amount of annotated training data. It is well known that the collection of annotated data is a time consuming and tedious task, which often cannot be performed sufficiently well for remote sensing tasks since the required data must cover a wide variety of scenes and objects. In this paper, we analyze the performance of such a network given a limited amount of training data and address the research question of whether artificially generated training data can be used to overcome the challenge of real-world data sets with a small amount of training data. For our experiments, we use the ISPRS 2D Semantic Labeling Contest Potsdam data set for vehicle detection, where we derive object-bounding boxes of vehicles suitable for our task. We generate artificial data based on vehicle blueprints and show that networks trained only on generated data may have a lower performance, but are still able to detect most of the vehicles found in the real data set. Moreover, we show that adding generated data to real-world data sets with a limited amount of training data, the performance can be increased significantly, and in some cases, almost reach baseline performance levels.

Short-wave infrared (SWIR) imaging systems with unmanned aerial vehicles (UAVs) are rarely used for remote sensing applications, like for vegetation monitoring. The reasons are that in the past, sensor systems covering the SWIR range were too expensive, too heavy, or not performing well enough, as, in contrast, it is the case in the visible and near-infrared range (VNIR). Therefore, our main objective is the development of a novel modular two-channel multispectral imaging system with a broad spectral sensitivity from the visible to the short-wave infrared spectrum (approx. 400 nm to 1700 nm) that is compact, lightweight and energy-efficient enough for UAV-based remote sensing applications. Various established vegetation indices (VIs) for mapping vegetation traits can then be set up by selecting any suitable filter combination. The study describes the selection of the individual components, starting with suitable camera modules, the optical as well as the control and storage parts. Special bandpass filters are used to select the desired wavelengths to be captured. A unique flange system has been developed, which also allows the filters to be interchanged quickly in order to adapt the system to a new application in a short time. The characterization of the system was performed in the laboratory with an integrating sphere and a climatic chamber. Finally, the integration of the novel modular VNIR/SWIR imaging system into a UAV and a subsequent first outdoor test flight, in which the functionality was tested, are described.

This study presents the results from a combined aerial survey performed with a hexacopter and a gyrocopter over a part of the Elbe estuary near Hamburg, Germany. The survey was conducted by the Federal Institute of Hydrology, Germany, and the Fraunhofer Application Center for Multimodal and Airborne Sensors as well as by a contracted engineering company with the aim to acquire spatial thermal infrared (TIR) data of the Hahnofer Nebenelbe, a branch of the Elbe estuary. Additionally, RGB and NIR data was captured to facilitate the identification of water surfaces and intertidal mudflats. The temperature distribution of the Elbe estuary affects all biological processes and in consequence the oxygen content, which is a key parameter in water quality. The oxygen levels vary in space between the main fairway and side channels. So far, only point measurements are available for monitoring and calibration/validation of water quality models. To better represent this highly dynamic system with a high spatial and temporal variability, tidal streams, heating and cooling, diffusion and mixing processes, spatially distributed data from several points of time within the tidal cycle are necessary. The data acquisition took place during two tidal cycles over two subsequent days in the summer of 2015. While the piloted gyrocopter covered the whole Hahnofer Nebenelbe seven times, the unmanned hexacopter covered a smaller section of the branch and tidal mudflats with a higher spatial and temporal resolution (16 coverages of the subarea). The gyrocopter data was acquired with a thermal imaging system and processed and georeferenced using the structure from motion algorithm with GPS information from the gyrocopter and optional ground control points. The hexacopter data was referenced based on ground control points and the GPS and position information of the acquisition system. Both datasets from the gyrocopter and the hexacopter are corrected for the effects of the atmosphere and emissivity of the water surface and compared to in situ measurements, taken during the data acquisition. Of particular interest is the effect of the observation angle on the brightness temperature acquired by the wide angle lenses on the platforms, which is up to 40 at the margins of the imagery. Here, both datasets show deviating temperatures, which are probably not due to actual temperature differences. We will discuss the position accuracy achieved over the water areas, the adaptation of atmospheric and emissivity correction to the observation angle and subsequent improvement of the temperature data. With two datasets of the same research area at different resolutions we will investigate the effects of the acquisition platforms, acquisition system and resolutions on the accuracy of the remotely sensed temperatures as well as their ability to represent temperature patterns of tidal currents and mixing processes.

In this paper the development of a lightweight and highly modularized airborne sensor platform for remote sensing applications utilizing a gyrocopter as a carrier platform is described. The current sensor configuration consists of a high resolution DSLR camera for VIS-RGB recordings. As a second sensor modality, a snapshot hyperspectral camera was integrated in the aircraft. Moreover a custom-developed thermal imaging system composed of a VIS-PAN camera and a LWIR-camera is used for aerial recordings in the thermal infrared range. Furthermore another custom-developed highly flexible imaging system for high resolution multispectral image acquisition with up to six spectral bands in the VIS-NIR range is presented. The performance of the overall system was tested during several flights with all sensor modalities and the precalculated demands with respect to spatial resolution and reliability were validated. The collected data sets were georeferenced, georectified, orthorectified and then stitched to mosaics.

Research and monitoring in fields like hydrology and agriculture are applications of airborne thermal infrared (TIR) cameras, which suffer from low spatial resolution and low quality lenses. Common ground control points (GCPs), lacking thermal activity and being relatively small in size, cannot be used in TIR images. Precise georeferencing and mosaicing however is necessary for data analysis. Adding a high resolution visible light camera (VIS) with a high quality lens very close to the TIR camera, in the same stabilized rig, allows us to do accurate geoprocessing with standard GCPs after fusing both images (VIS+TIR) using standard image registration methods.

Investigating human motion with expensive and accurate optical marker based systems has been the state of the art since long ago. However, markerless low-cost systems have always been a desideratum in the field of biomechanics and sports science. Due to increasing computer chip power and the corresponding progress in image processing techniques the realization of such a system has become feasible. With the advent of the Microsoft Kinect sensor in 2011 a flexible low-cost tool has entered the computer game market that enables markerless tracking of human motion. At first sight the Kinect provides an amazing accuracy. The goal of the present work is to quantitatively investigate the tracking accuracy of the Kinect sensor by studying the human gait cycle on a treadmill. The Kinect results are compared with data stemming from a VICON system which is regarded as a kind of gold standard in terms of spatial resolution. Subsequently, a post processing step is applied to the Kinect data using anthropometric data as a priori information in order to enhance Kinect’s tracking results.