Food Security is a crucial global concern and a key pillar of the Sustainable Development Goals (SDG 2 - Zero Hunger). To monitor food insecurity situations, several early warning systems are active today, driven by agencies such as FAO and WFP. These systems use a limited set of data types, e.g., agroclimatic data and indicators extracted from household surveys. Household surveys are expensive and time-consuming, and by consequence not frequent, which ultimately limits the ability to provide timely responses in vulnerable regions. In this work, we present a deep learning (DL) framework able to classify regional food security levels into three main categories i.e., poor, borderline and acceptable. The idea is to obtain such predictions by using openly accessible heterogeneous data sources, including satellite imagery, climate variables, and socioeconomic indicators. The proposed architecture employs multi-branch neural network models—Convolutional Neural Networks (CNNs) for spatial rasters (population density, land use/land cover maps), Recurrent Neural Networks for time series (e.g., rainfall, temperature, commodity prices), and Random Forests for feature fusion that integrates representations extracted from the two DL branches with a set of conjunctural variables. We validate our approach on two African countries, Burkina Faso and Rwanda, achieving classification accuracies exceeding 85

Applying deep learning to images of cropping systems provides new knowledge and insights in research and commercial applications. Semantic segmentation or pixel-wise classification, of RGB images acquired at the ground level, into vegetation and background is a critical step in the estimation of several canopy traits. Current state of the art methodologies based on convolutional neural networks (CNNs) are trained on datasets acquired under controlled or indoor environments. These models are unable to generalize to real-world images and hence need to be fine-tuned using new labelled datasets. This motivated the creation of the VegAnn - Vegetation Annotation - dataset, a collection of 3775 multi-crop RGB images acquired for different phenological stages using different systems and platforms in diverse illumination conditions. We anticipate that VegAnn will help improving segmentation algorithm performances, facilitate benchmarking and promote large-scale crop vegetation segmentation research.

Head (panicle) density is a major component in understanding crop yield, especially in crops that produce variable numbers of tillers such as sorghum and wheat. Use of panicle density both in plant breeding and in the agronomy scouting of commercial crops typically relies on manual counts observation, which is an inefficient and tedious process. Because of the easy availability of red–green–blue images, machine learning approaches have been applied to replacing manual counting. However, much of this research focuses on detection per se in limited testing conditions and does not provide a general protocol to utilize deep-learning-based counting. In this paper, we provide a comprehensive pipeline from data collection to model deployment in deep-learning-assisted panicle yield estimation for sorghum. This pipeline provides a basis from data collection and model training, to model validation and model deployment in commercial fields. Accurate model training is the foundation of the pipeline. However, in natural environments, the deployment dataset is frequently different from the training data (domain shift) causing the model to fail, so a robust model is essential to build a reliable solution. Although we demonstrate our pipeline in a sorghum field, the pipeline can be generalized to other grain species. Our pipeline provides a high-resolution head density map that can be utilized for diagnosis of agronomic variability within a field, in a pipeline built without commercial software.

The GEOGLAM crop monitor for early warning is based on the integration of the crop conditions assessments produced by regional systems. Discrepancies between these assessments can occur and are generally attributed to the interpretation of the vegetation and climate data. The premise of this article is that other sources of discrepancy related to the data themselves must also be considered. We conducted a comparative experiment of the growth vegetation anomalies routinely produced by four operational crop monitoring systems in West Africa [FEWSNET, GIEWS, ASAP, VAM] for the 2010–2020 period. We collected a set of normalized differences vegetation index-based indicators (% mean, % median, and Z-score) and proposed original methods to analyze and compare the spatio-temporal variations of these indices using Hovmöller representation, statistics, and spatial analysis. To facilitate systems comparison, a classification scheme based on the percentile rank values of anomaly indicators was applied to produce 3-class alarm maps (negative, absence, and positive anomalies). Results show that, on an annual basis, the per-pixel similarity is relatively low between the four systems [24.5%–34.1%], and that VAM and ASAP are the most similar (70%). The reasons of the products discrepancies come mainly from different preprocessing methods, especially the choice of the reference period used to calculate the anomaly. The negative alarm agreement classes show no eco-climatic zoning influence, but negative alarms hot-spots were locally observed. The negative alarm agreement maps can be a useful tool for early warning as they synthesize the information provided by the different systems, with a confidence level.

Pixel segmentation of high-resolution RGB images into chlorophyll-active or nonactive vegetation classes is a first step often required before estimating key traits of interest. We have developed the SegVeg approach for semantic segmentation of RGB images into three classes (background, green, and senescent vegetation). This is achieved in two steps: A U-net model is first trained on a very large dataset to separate whole vegetation from background. The green and senescent vegetation pixels are then separated using SVM, a shallow machine learning technique, trained over a selection of pixels extracted from images. The performances of the SegVeg approach is then compared to a 3-class U-net model trained using weak supervision over RGB images segmented with SegVeg as groundtruth masks. Results show that the SegVeg approach allows to segment accurately the three classes. However, some confusion is observed mainly between the background and senescent vegetation, particularly over the dark and bright regions of the images. The U-net model achieves similar performances, with slight degradation over the green vegetation: the SVM pixel-based approach provides more precise delineation of the green and senescent patches as compared to the convolutional nature of U-net. The use of the components of several color spaces allows to better classify the vegetation pixels into green and senescent. Finally, the models are used to predict the fraction of three classes over whole images or regularly spaced grid-pixels. Results show that green fraction is very well estimated (R2 = 0.94) by the SegVeg model, while the senescent and background fractions show slightly degraded performances (R2 = 0.70 and 0.73, respectively) with a mean 95% confidence error interval of 2.7% and 2.1% for the senescent vegetation and background, versus 1% for green vegetation. We have made SegVeg publicly available as a ready-to-use script and model, along with the entire annotated grid-pixels dataset. We thus hope to render segmentation accessible to a broad audience by requiring neither manual annotation nor knowledge or, at least, offering a pretrained model for more specific use.

Several crops bear reproductive organs (RO) at the top of the canopy after the flowering stage, such as ears for wheat, tassels for maize, and heads for sunflowers. RO present specific architecture and optical properties as compared to leaves and stems, which may impact canopy reflectance. This study aims to understand and quantify the influence of RO on the bi-directional variation of canopy reflectance and NDVI. Multispectral camera observations from a UAV were completed over wheat, maize, and sunflower just after flowering when the RO are fully developed and the leaf layer with only marginal senescence. The flights were designed to sample the BRDF with view zenith angles spanning from nadir to 60?and many compass directions. Three flights corresponding to three sun positions were completed under clear sly conditions. The camera was always pointing to two adjacent plots of few tenths of square meters: the RO were manually removed on one plot, while the other plot was kept undisturbed. Results showed that the three visible bands (450 nm, 570 nm, 675 nm), and in a lesser way the red edge band (730 nm) were strongly correlated. We, therefore, focused on the 675 nm and 850 nm bands. The Bi-Directional Reflectance (BRF) of the canopy without RO shows that the BRF values were almost symmetrical across the principal plane, even for maize and sunflower canopies with a strong row structure. Examination of the BRF difference between the canopy with and without RO indicate that the RO impact canopy BRDF for the three crops. The magnitude of the impacts depends on crop, wavelength and observational geometry. These observations are generally consistent with realistic 3D reflectance simulations. However, some discrepancies were noticed, mainly explained by the small magnitude of the RO effect on canopy BRF, and the approximations made when simulating the RO layer and its coupling with the bottom canopy layer. We finally demonstrated that the RO layer impact the estimates of canopy traits such as GAI as derived from the multispectral observations.

The Global Wheat Head Detection (GWHD) dataset was created in 2020 and has assembled 193,634 labelled wheat heads from 4,700 RGB images acquired from various acquisition platforms and 7 countries/institutions. With an associated competition hosted in Kaggle, GWHD has successfully attracted attention from both the computer vision and agricultural science communities. From this first experience in 2020, a few avenues for improvements have been identified, especially from the perspective of data size, head diversity and label reliability. To address these issues, the 2020 dataset has been reexamined, relabeled, and augmented by adding 1,722 images from 5 additional countries, allowing for 81,553 additional wheat heads to be added. We would hence like to release a new version of the Global Wheat Head Detection (GWHD) dataset in 2021, which is bigger, more diverse, and less noisy than the 2020 version. The GWHD 2021 is now publicly available at http://www.global-wheat.com/ and a new data challenge has been organized on AIcrowd to make use of this updated dataset.

Multispectral observations from unmanned aerial vehicles (UAVs) are currently used for precision agriculture and crop phenotyping applications to monitor a series of traits allowing the characterization of the vegetation status. However, the limited autonomy of UAVs makes the completion of flights difficult when sampling large areas. Increasing the throughput of data acquisition while not degrading the ground sample distance (GSD) is, therefore, a critical issue to be solved. We propose here a new image acquisition configuration based on the combination of two focal length (f) optics: an optics with f=4.2 mm is added to the standard f=8 mm (SS: single swath) of the multispectral camera (DS: double swath, double of the standard one). Two flights were completed consecutively in 2018 over a maize field using the AIRPHEN multispectral camera at 52 m altitude. The DS flight plan was designed to get 80% overlap with the 4.2 mm optics, while the SS one was designed to get 80% overlap with the 8 mm optics. As a result, the time required to cover the same area is halved for the DS as compared to the SS. The georeferencing accuracy was improved for the DS configuration, particularly for the Z dimension due to the larger view angles available with the small focal length optics. Application to plant height estimates demonstrates that the DS configuration provides similar results as the SS one. However, for both the DS and SS configurations, degrading the quality level used to generate the 3D point cloud significantly decreases the plant height estimates.

Deep learning based detection of sorghum panicles has been proposed to replace manual counting in field trials. However, model performance is highly sensitive to domain shift between training datasets associated with differences in genotypes, field conditions, and various lighting conditions. As labelling such datasets is expensive and laborious, we propose a pipeline of Contrastive Unpaired Translation (CUT) based domain adaptation method to improve detection performance in new datasets, including for completely different crop species. Firstly, original dataset is translated to other styles using CUT trained on unlabelled datasets from other domains. Then labels are corrected after synthesis of the new domain dataset. Finally, detectors are retrained on the synthesized dataset. Experiments show that, in case of sorghum panicles, the accuracy of the models when trained with synthetic images improve by fifteen to twenty percent. Furthermore, the models are more robust towards change in prediction thresholds. Hence, demonstrating the effectiveness of the pipeline.

The Global Wheat Head Detection (GWHD) dataset was created in 2020 and has assembled 193,634 labelled wheat heads from 4,700 RGB images acquired from various acquisition platforms and 7 countries/institutions. With an associated competition hosted in Kaggle, GWHD has successfully attracted attention from both the computer vision and agricultural science communities. From this first experience in 2020, a few avenues for improvements have been identified, especially from the perspective of data size, head diversity and label reliability. To address these issues, the 2020 dataset has been reexamined, relabeled, and augmented by adding 1,722 images from 5 additional countries, allowing for 81,553 additional wheat heads to be added. We now release a new version of the Global Wheat Head Detection (GWHD) dataset in 2021, which is bigger, more diverse, and less noisy than the 2020 version. The GWHD 2021 is now publicly available at http://www.global-wheat.com/ and a new data challenge has been organized on AIcrowd to make use of this updated dataset.

Early-stage plant density is an essential trait that determines the fate of a genotype under given environmental conditions and management practices. The use of RGB images taken from UAVs may replace the traditional visual counting in fields with improved throughput, accuracy, and access to plant localization. However, high-resolution images are required to detect the small plants present at the early stages. This study explores the impact of image ground sampling distance (GSD) on the performances of maize plant detection at three-to-five leaves stage using Faster-RCNN object detection algorithm. Data collected at high resolution (GSD ≈ 0.3 cm) over six contrasted sites were used for model training. Two additional sites with images acquired both at high and low (GSD ≈ 0.6 cm) resolutions were used to evaluate the model performances. Results show that Faster-RCNN achieved very good plant detection and counting (rRMSE = 0.08) performances when native high-resolution images are used both for training and validation. Similarly, good performances were observed (rRMSE = 0.11) when the model is trained over synthetic low-resolution images obtained by downsampling the native training high-resolution images and applied to the synthetic low-resolution validation images. Conversely, poor performances are obtained when the model is trained on a given spatial resolution and applied to another spatial resolution. Training on a mix of high- and low-resolution images allows to get very good performances on the native high-resolution (rRMSE = 0.06) and synthetic low-resolution (rRMSE = 0.10) images. However, very low performances are still observed over the native low-resolution images (rRMSE = 0.48), mainly due to the poor quality of the native low-resolution images. Finally, an advanced super resolution method based on GAN (generative adversarial network) that introduces additional textural information derived from the native high-resolution images was applied to the native low-resolution validation images. Results show some significant improvement (rRMSE = 0.22) compared to bicubic upsampling approach, while still far below the performances achieved over the native high-resolution images.

The Global Wheat Head Detection (GWHD) dataset was created in 2020 and has assembled 193,634 labelled wheat heads from 4700 RGB images acquired from various acquisition platforms and 7 countries/institutions. With an associated competition hosted in Kaggle, GWHD_2020 has successfully attracted attention from both the computer vision and agricultural science communities. From this first experience, a few avenues for improvements have been identified regarding data size, head diversity, and label reliability. To address these issues, the 2020 dataset has been reexamined, relabeled, and complemented by adding 1722 images from 5 additional countries, allowing for 81,553 additional wheat heads. We now release in 2021 a new version of the Global Wheat Head Detection dataset, which is bigger, more diverse, and less noisy than the GWHD_2020 version.

The detection of wheat heads in plant images is an important task for estimating pertinent wheat traits including head population density and head characteristics such as health, size, maturity stage, and the presence of awns. Several studies have developed methods for wheat head detection from high-resolution RGB imagery based on machine learning algorithms. However, these methods have generally been calibrated and validated on limited datasets. High variability in observational conditions, genotypic differences, development stages, and head orientation makes wheat head detection a challenge for computer vision. Further, possible blurring due to motion or wind and overlap between heads for dense populations make this task even more complex. Through a joint international collaborative effort, we have built a large, diverse, and well-labelled dataset of wheat images, called the Global Wheat Head Detection (GWHD) dataset. It contains 4700 high-resolution RGB images and 190000 labelled wheat heads collected from several countries around the world at different growth stages with a wide range of genotypes. Guidelines for image acquisition, associating minimum metadata to respect FAIR principles, and consistent head labelling methods are proposed when developing new head detection datasets. The GWHD dataset is publicly available at http://www.global-wheat.com/and aimed at developing and benchmarking methods for wheat head detection.

Accurate and timely observations of wheat phenology and, particularly, of heading date are instrumental for many scientific and technical domains such as wheat ecophysiology, crop breeding, crop management or precision agriculture. Visual annotation of the heading date in situ is a labour-intensive task that may become prohibitive in scientific and technical activities where high-throughput is needed. This study presents an automatic method to estimate wheat heading date from a series of daily images acquired by a fixed RGB camera in the field. A convolutional neural network (CNN) is trained to identify the presence of spikes in small patches. The heading date is then estimated from the dynamics of the spike presence in the patches over time. The method is applied and validated over a large set of 47 experimental sites located in different regions in France, covering three years with nine wheat cultivars. Results show that our method provides good estimates of the heading dates with a root mean square error close to 2 days when compared to the visual scoring from experts. It outperforms the predictions of a phenological model based on the ARCWHEAT crop model calibrated for our local conditions. The potentials and limits of the proposed methodology towards a possible operational implementation in agronomic applications and decision support systems are finally further discussed.

SENTINEL-2 observations are particularly well suited for crop management due to the short revisiting time and decametric spatial resolution. In this context, Green Area Index (GAI), the fraction of Photosynthetically Active Radiation Intercepted by the green vegetation (FIPAR), the fractional green vegetation cover (GF(0)) and Leaf Chlorophyll Content (LCC), are four variables of particular interest since they relate to the canopy state and can eventually be assimilated into crop functioning models to predict crop yield and optimize management practices. We present the results from the first year of the P2S2 project (Produits Pour Sentinel-2) that consisted in acquiring an extensive calibration and validation dataset over agricultural areas based on DHP measurements for GAI, GF(0) and FIPAR combined with SPAD-502 Minolta chlorophyll meter acquisitions for LCC.

Abstract Background Grain yield of wheat is greatly associated with the population of wheat spikes, i.e., $$spike~number~\text {m}^{-2}$$spikenumberm-2. To obtain this index in a reliable and efficient way, it is necessary to count wheat spikes accurately and automatically. Currently computer vision technologies have shown great potential to automate this task effectively in a low-end manner. In particular, counting wheat spikes is a typical visual counting problem, which is substantially studied under the name of object counting in Computer Vision. TasselNet, which represents one of the state-of-the-art counting approaches, is a convolutional neural network-based local regression model, and currently benchmarks the best record on counting maize tassels. However, when applying TasselNet to wheat spikes, it cannot predict accurate counts when spikes partially present. Results In this paper, we make an important observation that the counting performance of local regression networks can be significantly improved via adding visual context to the local patches. Meanwhile, such context can be treated as part of the receptive field without increasing the model capacity. We thus propose a simple yet effective contextual extension of TasselNet—TasselNetv2. If implementing TasselNetv2 in a fully convolutional form, both training and inference can be greatly sped up by reducing redundant computations. In particular, we collected and labeled a large-scale wheat spikes counting (WSC) dataset, with 1764 high-resolution images and 675,322 manually-annotated instances. Extensive experiments show that, TasselNetv2 not only achieves state-of-the-art performance on the WSC dataset ($$91.01\%$$91.01% counting accuracy) but also is more than an order of magnitude faster than TasselNet (13.82 fps on $$912\times 1216$$912×1216 images). The generality of TasselNetv2 is further demonstrated by advancing the state of the art on both the Maize Tassels Counting and ShanghaiTech Crowd Counting datasets. Conclusions This paper describes TasselNetv2 for counting wheat spikes, which simultaneously addresses two important use cases in plant counting: improving the counting accuracy without increasing model capacity, and improving efficiency without sacrificing accuracy. It is promising to be deployed in a real-time system with high-throughput demand. In particular, TasselNetv2 can achieve sufficiently accurate results when training from scratch with small networks, and adopting larger pre-trained networks can further boost accuracy. In practice, one can trade off the performance and efficiency according to certain application scenarios. Code and models are made available at: https://tinyurl.com/TasselNetv2.

Wheat ear density estimation is an appealing trait for plant breeders. Current manual counting is tedious and inefficient. In this study we investigated the potential of convolutional neural networks (CNNs) to provide accurate ear density using nadir high spatial resolution RGB images. Two different approaches were investigated, either using the Faster-RCNN state-of-the-art object detector or with the TasselNet local count regression network. Both approaches performed very well (rRMSE approximate to 6%) when applied over the same conditions as those prevailing for the calibration of the models. However, Faster-RCNN was more robust when applied to a dataset acquired at a later stage with ears and background showing a different aspect because of the higher maturity of the plants. Optimal spatial resolution for Faster-RCNN was around 0.3 mm allowing to acquire RGB images from a UAV platform for high-throughput phenotyping of large experiments. Comparison of the estimated ear density with in-situ manual counting shows reasonable agreement considering the relatively small sampling area used for both methods. Faster-RCNN and in-situ counting had high and similar heritability (H-2 approximate to 85%), demonstrating that ear density derived from high resolution RGB imagery could replace the traditional counting method.

Total above-ground biomass at harvest and ear density are two important traits that characterize wheat genotypes. Two experiments were carried out in two different sites where several genotypes were grown under contrasted irrigation and nitrogen treatments. A high spatial resolution RGB camera was used to capture the residual stems standing straight after the cutting by the combine machine during harvest. It provided a ground spatial resolution better than 0.2 mm. A Faster Regional Convolutional Neural Network (Faster-RCNN) deep-learning model was first trained to identify the stems cross section. Results showed that the identification provided precision and recall close to 95%. Further, the balance between precision and recall allowed getting accurate estimates of the stem density with a relative RMSE close to 7% and robustness across the two experimental sites. The estimated stem density was also compared with the ear density measured in the field with traditional methods. A very high correlation was found with almost no bias, indicating that the stem density could be a good proxy of the ear density. The heritability/repeatability evaluated over 16 genotypes in one of the two experiments was slightly higher (80%) than that of the ear density (78%). The diameter of each stem was computed from the profile of gray values in the extracts of the stem cross section. Results show that the stem diameters follow a gamma distribution over each microplot with an average diameter close to 2.0 mm. Finally, the biovolume computed as the product of the average stem diameter, the stem density, and plant height is closely related to the above-ground biomass at harvest with a relative RMSE of 6%. Possible limitations of the findings and future applications are finally discussed.

Phenotyping in breeding trials is the basis for the selection of new varieties of food, feed, and fiber crops that support the continued growth of world population. In addition to economic yield, breeders measure many phenotypes (also called traits) associated with the adaptation of these crops, such as the time of flowering, the height of the crop canopy, and the development of the canopy as it grows to maximum size and then senesces late in the season. The rapidly decreasing costs, and the convenience of use of UAVs (unmanned aerial vehicles) is providing plant breeders with new tools with which to estimate some of the traits that are traditionally measured. With appropriate sets of measurements, it is possible to also estimate more complex traits, for example, the radiation use efficiency (RUE) of a crop can be estimated when it is possible to track the change in light interception over time. Visual, thermal, and multispectral cameras are key tools in monitoring crops by UAV, with LIDAR and hyperspectral instruments starting to come into use as they become sufficiently miniaturized. In this chapter we outline the use of these types of cameras in the characterization of plant phenotypes that assist breeders in the selection of genotypes, ideally at early stages of the breeding program. We outline the hierarchy of data values as they are transformed from raw data (L0) through calibrated normalized quantities (L1) to state variables (L2) and eventually to functional traits (L3). Phenotypes that are directly observed by breeders are usually L2 traits, while L3 are derived traits, such as RUE, which are not directly measured by a sensor. We describe a workflow for managing and analysis of UAV-captured imagery, and consider issues related to pixel resolution and camera parameters and the need for “local” calibration approaches whereby a trait may be manually measured on a subset of plots, while being measured by UAV, in order to derive a predictive relationship from the subset to the entire trial. In the remaining part of the chapter, we provide examples and suggestions for improvements from our own research, based on the types of traits that are measured at the early, mid and late stages of the season as related to plant development, canopy cover and morphological traits, and segmentation of objects and spatial variation in signal intensity. Plant breeding needs to be accelerated in order to keep up with population growth and changes in climate. Phenomics being developed with proximal sensing tools contribute to these accelerated breeding methods such as genomic selection, and this chapter attempts to provide some guidelines for experts in remote sensing to engage in this area of research.