IntroductionComputer vision and deep learning (DL) techniques have succeeded in a wide range of diverse fields. Recently, these techniques have been successfully deployed in plant science applications to address food security, productivity, and environmental sustainability problems for a growing global population. However, training these DL models often necessitates the large-scale manual annotation of data which frequently becomes a tedious and time-and-resource- intensive process. Recent advances in self-supervised learning (SSL) methods have proven instrumental in overcoming these obstacles, using purely unlabeled datasets to pre-train DL models.MethodsHere, we implement the popular self-supervised contrastive learning methods of NNCLR Nearest neighbor Contrastive Learning of visual Representations) and SimCLR (Simple framework for Contrastive Learning of visual Representations) for the classification of spatial orientation and segmentation of embryos of maize kernels. Maize kernels are imaged using a commercial high-throughput imaging system. This image data is often used in multiple downstream applications across both production and breeding applications, for instance, sorting for oil content based on segmenting and quantifying the scutellum’s size and for classifying haploid and diploid kernels.Results and discussionWe show that in both classification and segmentation problems, SSL techniques outperform their purely supervised transfer learning-based counterparts and are significantly more annotation efficient. Additionally, we show that a single SSL pre-trained model can be efficiently finetuned for both classification and segmentation, indicating good transferability across multiple downstream applications. Segmentation models with SSL-pretrained backbones produce DICE similarity coefficients of 0.81, higher than the 0.78 and 0.73 of those with ImageNet-pretrained and randomly initialized backbones, respectively. We observe that finetuning classification and segmentation models on as little as 1% annotation produces competitive results. These results show SSL provides a meaningful step forward in data efficiency with agricultural deep learning and computer vision.

These are companion data and models of manuscript "Self-Supervised Maize Kernel Classification and Segmentation for Embryo Identification" that was submitted to Frontiers in Plant Science. The data is organized into three main folders: 'class_full_imgs', 'seg_full_imgs', and 'unlabeled'. The 'class_full_imgs' folder contains labeled data used to train the classification model, which is divided into train, validation, and test subfolders. Each of these subfolders contains 'oriented' and 'non-oriented' images. The 'seg_full_imgs' folder contains labeled data used to train the segmentation model. The 'InputImages' subfolder contains raw images, and the 'OutputImages' subfolder contains the segmented images. The 'unlabeled' folder contains images without any labels. These images were used for self-supervised pretraining of classification and segmentation models. The trained models can be found in Trained_models.zip. There are four zip files: - "simclr_pretrained_bb.zip" contains the selected pretrained backbone trained via SimCLR. - "nnclr_pretrained_bb.zip" contains the selected pretrained backbone trained via NNCLR. - "finetuned_classification" contains classification models which have undergone end-to-end finetuning, split into supervised and self-supervision-pretrained models. - "segmentation" contains image segmentation models, where the names refer to the pretraining method.

The cyber-agricultural system (CAS) represents an overarching framework of agriculture that leverages recent advances in ubiquitous sensing, artificial intelligence, smart actuators, and scalable cyberinfrastructure (CI) in both breeding and production agriculture. We discuss the recent progress and perspective of the three fundamental components of CAS - sensing, modeling, and actuation - and the emerging concept of agricultural digital twins (DTs). We also discuss how scalable CI is becoming a key enabler of smart agriculture. In this review we shed light on the significance of CAS in revolutionizing crop breeding and production by enhancing efficiency, productivity, sustainability, and resilience to changing climate. Finally, we identify underexplored and promising future directions for CAS research and development.

Abstract Insect pests cause significant damage to food production, so early detection and efficient mitigation strategies are crucial. There is a continual shift toward machine learning (ML)‐based approaches for automating agricultural pest detection. Although supervised learning has achieved remarkable progress in this regard, it is impeded by the need for significant expert involvement in labeling the data used for model training. This makes real‐world applications tedious and oftentimes infeasible. Recently, self‐supervised learning (SSL) approaches have provided a viable alternative to training ML models with minimal annotations. Here, we present an SSL approach to classify 22 insect pests. The framework was assessed on raw and segmented field‐captured images using three different SSL methods, Nearest Neighbor Contrastive Learning of Visual Representations (NNCLR), Bootstrap Your Own Latent, and Barlow Twins. SSL pre‐training was done on ResNet‐18 and ResNet‐50 models using all three SSL methods on the original RGB images and foreground segmented images. The performance of SSL pre‐training methods was evaluated using linear probing of SSL representations and end‐to‐end fine‐tuning approaches. The SSL‐pre‐trained convolutional neural network models were able to perform annotation‐efficient classification. NNCLR was the best performing SSL method for both linear and full model fine‐tuning. With just 5% annotated images, transfer learning with ImageNet initialization obtained 74% accuracy, whereas NNCLR achieved an improved classification accuracy of 79% for end‐to‐end fine‐tuning. Models created using SSL pre‐training consistently performed better, especially under very low annotation, and were robust to object class imbalances. These approaches help overcome annotation bottlenecks and are resource efficient.

Crop pest detection and mitigation remains an extremely challenging task for the farmers. Majority of the pest classification and detection techniques rely on supervised deep learning frameworks that require significant human intervention in labeling the input data, thereby making the down-stream tasks tedious. Therefore, this study presents a self-supervised learning (SSL) approach to classifying 12 types of agricultural insect pests from 9549 RGB images, by leveraging the Bootstrap your own latent (BYOL) algorithm. SSL uses minimal labeling and is indifferent to data augmentations or distortions. Hence, latent representations from pretrained SSL networks could be generalized well for downstream tasks like classification or object detection. For desirable classification of the insect images, the greatest challenges observed were: i) large intra-class variation (the same insect was found with different colors and patterns), and ii) complex background with inconspicuous foreground. Hence, to overcome these issues and aid generalizability of the representations learned through BYOL, entropy-guided segmentation (segments based on texture not color), is proposed as input to the SSL network in this study. Both raw and segmented images were separately fed to two independent BYOL SSL networks, i.e., with ResNet18 and ResNet50 architectures as the backbone. The efficacy of the latent representations for downstream applications was assessed using linear evaluation, and subsequently compared with traditional transfer learning outcomes from ResNet18 and ResNet50. The results indicated that, while ResNet50 backbone intuitively performed better in all cases, SSL aided with entropy-based segmentation led to ~94% classification accuracy compared to raw images (with ~90% maximum accuracy).

Deep neural networks are increasing being deployed for automating plant stress identification and quantification. However, as this area grows in importance, alleviating privacy concerns of practitioners becomes a major challenge. In this paper, we present a deep learning framework for plant stress phenotyping that guarantees privacy to both the owner of the data, as well as the developer of the model. Our framework leverages recent advances in deep neural network design for accelerated private inference (PI) using secure multiparty communication. We showcase our framework on a large-scale image dataset, demonstrate that our newly designed models enjoy nearly 2 orders of magnitude speedup in inference time over state-of-the-art baselines.

Abstract Deep learning (DL) methods have transformed the way we extract plant traits—both under laboratory as well as field conditions. Evidence suggests that “well‐trained” DL models can significantly simplify and accelerate trait extraction as well as expand the suite of extractable traits. Training a DL model typically requires the availability of copious amounts of annotated data; however, creating large‐scale annotated dataset requires nontrivial efforts, time, and resources. This limitation has become a major bottleneck in deploying DL tools in practice. Self‐supervised learning (SSL) methods give exciting solution to this problem, as these methods use unlabeled data to produce pretrained models for subsequent fine‐tuning on labeled data and have demonstrated superior transfer learning performance on down‐stream classification tasks. We investigated the application of SSL methods for plant stress classification using few labels. We select a plant stress classification problem to test the effectiveness of SSL, as it is a fundamentally challenging problem due to (a) disease classification which depends on the abnormalities in a small number of pixels, (b) high data imbalance across different classes, and (c) fewer annotated and available plant stress images than in other domains. We compared seven SSL approaches spanning four broad classes of SSL methods on soybean [Glycine max L. (Merr.)] plant stress dataset and report that pretraining on unlabeled plant stress images significantly outperforms transfer learning methods using random initialization for plant stress classification. In summary, SSL‐based model initialization and data curation improves annotation efficiency for plant stress classification tasks and will circumvent data annotation challenges associated with DL methods.

Plant stress phenotyping is essential to select stress-resistant varieties and develop better stress-management strategies. Standardization of visual assessments and deployment of imaging techniques have improved the accuracy and reliability of stress assessment in comparison with unaided visual measurement. The growing capabilities of machine learning (ML) methods in conjunction with image-based phenotyping can extract new insights from curated, annotated, and high-dimensional datasets across varied crops and stresses. We propose an overarching strategy for utilizing ML techniques that methodically enables the application of plant stress phenotyping at multiple scales across different types of stresses, program goals, and environments.

Globally, crop insect pests lead to 10 – 40% yield loss. However, crop insect pest detection and mitigation remain an extremely challenging task for the farmers, due to several factors. While supervised learning has achieved a remarkable feat in insect detection, it requires significant human intervention in labeling the input data, thereby making the downstream tasks tedious and sometimes infeasible. This is particularly the case for identifying insects in the field, where labeling is tedious. Here, we present a self-supervised learning (SSL) approach – Bootstrap your own latent (BYOL) to classify 12 types of agricultural insect pests using minimal labeling. Both raw and segmented images were separately fed to the BYOL SSL method, and the linear classification accuracies from the representations learned were examined. The results indicate that using segmented images as input to BYOL could lead up to 94% classification accuracy.

Soybean [Glycine max (L.) Merr.] breeders and geneticists routinely evaluate thousands of plots per year in order to characterize various accessions and breedingBreeding populations for a multitude of traits, for example, morphological, physiological, abiotic and biotic stressAbiotic and biotic stresses, plant organs, and seed composition. Most of these trait evaluations require experienced raters to spend countless hours, recording phenotypes for each genotype in different filial generations. Plant breeders strive to work with increased population sizes, and improved accuracy of selection to increase the response to selection. These requirements have motivated the development of high-throughput phenotypingHigh-Throughput Phenotyping (HTP) (HTP) methods and associated tools (i.e., hardware) and software solutions. This chapter consists of several topics related to HTP, including sensorsSensors, unmanned aerial systemsUnmanned Aerial Systems (UAS), and ground robotsGround robots, as these are important components of plant phenotyping in the new technological era. The advances in image-based analysisImage-based analysis and machine learningMachine learning methods have accompanied the improvements in phenotyping capabilities, both aerial and ground. This chapter includes the current state of the artState of the art in types of sensors, aerial, and ground-based HTP, in conjunction with machine learningMachine learning-based analytics, particularly for physiological and morphological traits, abiotic and biotic stressesAbiotic and biotic stresses, and root-related traits. Advances in the integrationIntegration of HTPHigh-Throughput Phenotyping (HTP) with crop modeling are provided. Finally, the complementary relationship between HTP and genomic studies is explained with pertinent examples.

Currently, automated canopy stress classification for field crops rely on single-perspective, two-dimensional (2D) photographs, typically top view imaging via UAV. However, plant stress symptoms may appear throughout the canopy, and a single viewpoint photograph may not capture the entire region affected by the stress. Recent developments in efficient, large scale, 3D point cloud capture of agricultural fields open up the possibility of more comprehensive stress identification and rating. We hypothesized that utilizing the 3D point cloud will allow multi-perspective construction of plant canopy, and subsequent training of more accurate plant stress identification and its rating in field. We utilize a RGB 3D point cloud of a field where a diversity panel of soybean under Iron Deficiency chlorosis (IDC) stress was grown. We explore both multiview projection as well as area-preserving map projection methods to obtain parameterized 2D images depicting the complete 3D canopy surface. This approach allowed us to create models agnostic to canopy size/shape, while allowing us to leverage pretrained deep learning models – trained on 2D image data. Our preliminary results are promising, and we continue to finetune these machine learning pipelines for classifying plant

Deep learning techniques have been successfully deployed for automating plant stress identification and quantification. In recent years, there is a growing push towards training models that are interpretable -i.e. that justify their classification decisions by visually highlighting image features that were crucial for classification decisions. The expectation is that trained network models utilize image features that mimic visual cues used by plant pathologists. In this work, we compare some of the most popular interpretability methods: Saliency Maps, SmoothGrad, Guided Backpropogation, Deep Taylor Decomposition, Integrated Gradients, Layer-wise Relevance Propagation and Gradient times Input, for interpreting the deep learning model. We train a DenseNet-121 network for the classification of eight different soybean stresses (biotic and abiotic). Using a dataset consisting of 16,573 RGB images of healthy and stressed soybean leaflets captured under controlled conditions, we obtained an overall classification accuracy of 95.05 \%. For a diverse subset of the test data, we compared the important features with those identified by a human expert. We observed that most interpretability methods identify the infected regions of the leaf as important features for some -- but not all -- of the correctly classified images. For some images, the output of the interpretability methods indicated that spurious feature correlations may have been used to correctly classify them. Although the output explanation maps of these interpretability methods may be different from each other for a given image, we advocate the use of these interpretability methods as `hypothesis generation' mechanisms that can drive scientific insight.

Premise Trichomes are hair‐like appendages extending from the plant epidermis. They serve many important biotic roles, including interference with herbivore movement. Characterizing the number, density, and distribution of trichomes can provide valuable insights on plant response to insect infestation and define the extent of plant defense capability. Automated trichome counting would speed up this research but poses several challenges, primarily because of the variability in coloration and the high occlusion of the trichomes. Methods and Results We developed a simplified method for image processing for automated and semi‐automated trichome counting. We illustrate this process using 30 leaves from 10 genotypes of soybean ( Glycine max ) differing in trichome abundance. We explored various heuristic image‐processing methods including thresholding and graph‐based algorithms to facilitate trichome counting. Of the two automated and two semi‐automated methods for trichome counting tested and with the help of regression analysis, the semi‐automated manually annotated trichome intersection curve method performed best, with an accuracy of close to 90% compared with the manually counted data. Conclusions We address trichome counting challenges including occlusion by combining image processing with human intervention to propose a semi‐automated method for trichome quantification. This provides new opportunities for the rapid and automated identification and quantification of trichomes, which has applications in a wide variety of disciplines.

Deep learning models have been successfully deployed for a diverse array of image-based plant phenotyping applications including disease detection and classification. However, successful deployment of supervised deep learning models requires large amount of labeled data, which is a significant challenge in plant science (and most biological) domains due to the inherent complexity. Specifically, data annotation is costly, laborious, time consuming and needs domain expertise for phenotyping tasks, especially for diseases. To overcome this challenge, active learning algorithms have been proposed that reduce the amount of labeling needed by deep learning models to achieve good predictive performance. Active learning methods adaptively select samples to annotate using an acquisition function to achieve maximum (classification) performance under a fixed labeling budget. We report the performance of four different active learning methods, (1) Deep Bayesian Active Learning (DBAL), (2) Entropy, (3) Least Confidence, and (4) Coreset, with conventional random sampling-based annotation for two different image-based classification datasets. The first image dataset consists of soybean [Glycine max L. (Merr.)] leaves belonging to eight different soybean stresses and a healthy class, and the second consists of nine different weed species from the field. For a fixed labeling budget, we observed that the classification performance of deep learning models with active learning-based acquisition strategies is better than random sampling-based acquisition for both datasets. The integration of active learning strategies for data annotation can help mitigate labelling challenges in the plant sciences applications particularly where deep domain knowledge is required.

BACKGROUND:Hyperspectral imaging is emerging as a promising approach for plant disease identification. The large and possibly redundant information contained in hyperspectral data cubes makes deep learning based identification of plant diseases a natural fit. Here, we deploy a novel 3D deep convolutional neural network (DCNN) that directly assimilates the hyperspectral data. Furthermore, we interrogate the learnt model to produce physiologically meaningful explanations. We focus on an economically important disease, charcoal rot, which is a soil borne fungal disease that affects the yield of soybean crops worldwide.RESULTS:Based on hyperspectral imaging of inoculated and mock-inoculated stem images, our 3D DCNN has a classification accuracy of 95.73% and an infected class F1 score of 0.87. Using the concept of a saliency map, we visualize the most sensitive pixel locations, and show that the spatial regions with visible disease symptoms are overwhelmingly chosen by the model for classification. We also find that the most sensitive wavelengths used by the model for classification are in the near infrared region (NIR), which is also the commonly used spectral range for determining the vegetative health of a plant.CONCLUSION:The use of an explainable deep learning model not only provides high accuracy, but also provides physiological insight into model predictions, thus generating confidence in model predictions. These explained predictions lend themselves for eventual use in precision agriculture and research application using automated phenotyping platforms.

The rate of advancement made in phenomic-assisted breeding methodologies has lagged those of genomic-assisted techniques, which is now a critical component of mainstream cultivar development pipelines. However, advancements made in phenotyping technologies have empowered plant scientists with affordable high-dimensional datasets to optimize the operational efficiencies of breeding programs. Phenomic and seed yield data was collected across six environments for a panel of 292 soybean accessions with varying genetic improvements. Random forest, a machine learning (ML) algorithm, was used to map complex relationships between phenomic traits and seed yield and prediction performance assessed using two cross-validation (CV) scenarios consistent with breeding challenges. To develop a prescriptive sensor package for future high-throughput phenotyping deployment to meet breeding objectives, feature importance in tandem with a genetic algorithm (GA) technique allowed selection of a subset of phenotypic traits, specifically optimal wavebands. The results illuminated the capability of fusing ML and optimization techniques to identify a suite of in-season phenomic traits that will allow breeding programs to decrease the dependence on resource-intensive end-season phenotyping (e.g., seed yield harvest). While we illustrate with soybean, this study establishes a template for deploying multitrait phenomic prediction that is easily amendable to any crop species and any breeding objective.

Since the beginning of domesticated agriculture, plants have been selected based on phenotypic characteristics visually observed above ground. Often roots and soil root interactions are less commonly studied and observed. Some breeders now are starting to investigate beyond the generally known knowledge of root system architecture (RSA). Legume crops present a unique, often hidden relationship, underground. Historically, researchers and farmers rarely quantify the number of nodules present on a soybean root. Nodules are the structural component of a soybean root where Bradyrhizobium japonicum lives in a symbiotic relationship with a legume to fix nitrogen for the plant. For breeders to more effectively quantify and study this beneficial relationship, a new tool has been developed called SNAP?Soybean Nodule Acquisition Program?which uses computer vision and machine learning to identify and quantify the nodules in a 2D soybean root image

In this paper, we describe the Perceptual Image Restoration and Manipulation (PIRM) workshop challenge on spectral image super-resolution, motivate its structure and conclude on results obtained by the participants. The challenge is one of the first of its kind, aiming at leveraging modern machine learning techniques to achieve spectral image super-resolution. It comprises of two tracks. The first of these (Track 1) is about example-based single spectral image super-resolution. The second one (Track 2) is on colour-guided spectral image super-resolution. In this manner, Track 1 focuses on the problem of super-resolving the spatial resolution of spectral images given training pairs of low and high spatial resolution spectral images. Track 2, on the other hand, aims to leverage the inherently higher spatial resolution of colour (RGB) cameras and the link between spectral and trichromatic images of the scene. The challenge in both tracks is then to recover a super-resolved image making use of low-resolution imagery at the input. We also elaborate upon the methods used by the participants, summarise the results and discuss their rankings.

Our overarching goal is to develop an accurate and explainable model for plant disease identification using hyperspectral data. Charcoal rot is a soil borne fungal disease that affects the yield of soybean crops worldwide. Hyperspectral images were captured at 240 different wavelengths in the range of 383 - 1032 nm. We developed a 3D Convolutional Neural Network model for soybean charcoal rot disease identification. Our model has classification accuracy of 95.73\% and an infected class F1 score of 0.87. We infer the trained model using saliency map and visualize the most sensitive pixel locations that enable classification. The sensitivity of individual wavelengths for classification was also determined using the saliency map visualization. We identify the most sensitive wavelength as 733 nm using the saliency map visualization. Since the most sensitive wavelength is in the Near Infrared Region(700 - 1000 nm) of the electromagnetic spectrum, which is also the commonly used spectrum region for determining the vegetation health of the plant, we were more confident in the predictions using our model.

This paper reviews the first challenge on spectral image reconstruction from RGB images, i.e., the recovery of whole-scene hyperspectral (HS) information from a 3channel RGB image. The challenge was divided into 2 tracks: the "Clean" track sought HS recovery from noiseless RGB images obtained from a known response function (representing spectrally-calibrated camera) while the "Real World" track challenged participants to recover HS cubes from JPEG-compressed RGB images generated by an unknown response function. To facilitate the challenge, the BGU Hyperspectral Image Database [4] was extended to provide participants with 256 natural HS training images, and 5+ 10 additional images for validation and testing, respectively. The "Clean" and "Real World" tracks had 73 and 63 registered participants respectively, with 12 teams competing in the final testing phase. Proposed methods and their corresponding results are reported in this review.