The safe and reliable operation of autonomous agricultural vehicles requires an advanced environment perception system. An important component of perception systems is vision-based algorithms for detecting objects and other structures in the fields. This paper presents an ensemble method for combining outputs of three scene understanding tasks: semantic segmentation, object detection and anomaly detection in the agricultural context. The proposed framework uses an object detector to detect seven agriculture-specific classes. The anomaly detector detects all other objects that do not belong to these classes. In addition, the segmentation map of the field is utilized to provide additional information if the objects are located inside or outside the field area. The detections of different algorithms are combined at inference time, and the proposed ensemble method is independent of underlying algorithms. The results show that combining object detection with anomaly detection can increase the number of detected objects in agricultural scene images.

Vision-based object detection is essential for safe and efficient field operation for autonomous agricultural vehicles. However, one of the challenges in transferring state-of-the-art object detectors to the agricultural domain is the limited availability of labeled datasets. This paper seeks to address this challenge by utilizing two object detection models based on YOLOv5, one pre-trained on a large-scale dataset for detecting general classes of objects and one trained to detect a smaller number of agriculture-specific classes. To combine the detections of the models at inference, we propose an ensemble module based on a hierarchical structure of classes. Results show that applying the proposed ensemble module increases mAP@.5 from 0.575 to 0.65 on the test dataset and reduces the misclassification of similar classes detected by different models. Furthermore, by translating detections from base classes to a higher level in the class hierarchy, we can increase the overall mAP@.5 to 0.701 at the cost of reducing class granularity.

The safe in-field operation of autonomous agricultural vehicles requires detecting all objects that pose a risk of collision. Current vision-based algorithms for object detection and classification are unable to detect unknown classes of objects. In this paper, the problem is posed as anomaly detection instead, where convolutional autoencoders are applied to identify any objects deviating from the normal pattern. Training an autoencoder network to reconstruct normal patterns in agricultural fields makes it possible to detect unknown objects by high reconstruction error. Basic autoencoder (AE), vector-quantized variational autoencoder (VQ-VAE), denoising autoencoder (DAE) and semisupervised autoencoder (SSAE) with a max-margin-inspired loss function are investigated and compared with a baseline object detector based on YOLOv5. Results indicate that SSAE with an area under the curve for precision/recall (PR AUC) of 0.9353 outperforms other autoencoder models and is comparable to an object detector with a PR AUC of 0.9794. Qualitative results show that SSAE is capable of detecting unknown objects, whereas the object detector is unable to do so and fails to identify known classes of objects in specific cases.

Automated and robotic ground-vehicle solutions are gradually becoming part of the agricultural industry, where they are used for performing tasks such as feeding, herding, planting, harvesting, and weed spraying. Agricultural machinery operates in both indoor and outdoor farm environments, resulting in changing operational conditions. Variation in the load transported by ground-vehicles is a common occurrence in the agricultural domain, in tasks such as animal feeding and field spraying. The development of automated and robotic ground-vehicle solutions for conditions and scenarios in the agricultural domain is a complex task, which requires input from multiple engineering disciplines. This PhD thesis proposes modelling and simulation for the research and development of automated and robotic ground-vehicle solutions for purposes such as component development, virtual prototype testing, and scenario evaluation. The collaboration of multiple engineering disciplines is achieved by combining multiple modelling and simulation tools from different engineering disciplines. These combined models are known as co-models and their execution is referred to as co-simulation. The results of this thesis are a model-based development methodology for automated and robotic ground-vehicles utilised for a number of research and development cases. The co-models of the automated and robotic ground vehicles were created using the model-based development methodology, and they contribute to the future development support in this research domain. The thesis presents four contributions toward the exploration of a chosen design space for an automated or robotic ground vehicle. Solutions obtained using co-modelling and co-simulation are deployed to their ground-vehicle realisations, which ensures that all stages of development are covered.

Unmanned Aerial Systems (UAS) operations are evolving towards autonomous flight beyond visual line of sight (BVLOS), which requires moving assessments normally conducted by the remote pilot to the autonomous software. An important assessment currently conducted by the remote pilot is whether the current weather conditions pose a safety risk to the flight. This work deals with the development of a software framework for analysing weather data. The developed framework is capable of providing a weather analysis report to a remote supervisor or to an autonomous decision-making software. The software framework has been tested using historical weather data for a full calendar year provided by IBM. The weather data was applied to four different UAS ranging from small UAS to a passenger UAS. The results obtained show that during a full year, flight was possible from 53.9 % to 95.8 % of the time depending on the UAS. We consider the software framework an important step towards improving the operational safety for autonomous UAS operations under BVLOS conditions.

ESA provides Sentinel-1 synthetic aperture radar satellite data freely for research and industry. The Sentinel-1 data have shown the potential for remotely monitoring conditions in individual agricultural fields on a weekly basis. Researchers have access to the same Sentinel-1 dataset, so independent validation should be possible. Well documented studies performed with Sentinel-1 will allow other researchers the ability to reproduce the experiments and either validate or repudiate the presented findings. Based on the current state-of-the-art study, a web service was provided for the agricultural domain, which can be downloaded freely from Github. The running web service provides the ability to monitor local conditions by using the recorded Sentinel-1 information, combined with a priori knowledge from broad acre fields. Correlating the Sentinel-1 data to actual conditions related to a specific application is still a task that individual researchers must perform to utilize the service. In this paper, we present our methodology for converting the Sentinel-1 data to a form that is more accessible to researchers in the agricultural domain. Therefore, the goal of the current study was to make the Sentinel-1 data available efficiently, so that the experts in this field can focus on correlating and comparing it to reference data and measurements collected in the field. The function of the web service is illustrated with concrete application examples in the agricultural domain.

This paper describes a collaborative modelling approach to automated and robotic agricultural vehicle design. The Cresendo technology allows engineers from different disciplines to collaborate and produce system models. The combined models are called co-models and their execution co-simulation. To support future development efforts a template library of different vehicle and controllers types are provided. This paper describes a methodology to developing co-models from initial problem definition to deployment of the actual system. We illustrate the development methodology with an example development case from the agricultural domain. The case relates to an encountered speed controller problem on a differential driven vehicle, where we iterate through different candidate solutions and end up with an adaptive controller solution based on a combination of classical control and learning feedforward. The second case is an example of combining human control interface and co-simulation of agricultural robotic operation to illustrate collaborative development

ESA operates the Sentinel-1 satellites, which provides Synthetic Aperture Radar (SAR) data of Earth. Recorded Sentinel-1 data have shown a potential for remotely observing and monitoring local conditions on broad acre fields. Remote sensing using Sentinel-1 have the potential to provide daily updates on the current conditions in the individual fields and at the same time give an overview of the agricultural areas in the region. Research depends on the ability of independent validation of the presented results. In the case of the Sentinel-1 satellites, every researcher has access to the same base dataset, and therefore independent validation is possible. Well documented research performed with Sentinel-1 allow other research the ability to redo the experiments and either validate or falsify presented findings. Based on current state-of-art research we have chosen to provide a service for researchers in the agricultural domain. The service allows researchers the ability to monitor local conditions by using the Sentinel-1 information combined with a priori knowledge from broad acre fields. Correlating processed Sentinel-1 to the actual conditions is still a task the individual researchers must perform to benefit from the service. In this paper, we presented our methodology in translating sentinel-1 data to a level that is more accessible to researchers in the agricultural field. The goal here was to make the data more easily available, so the primary focus can be on correlating and comparing to measurements collected in the broadacre fields. We illustrate the value of the service with three examples of the possible application areas. The presented application examples are all based on Denmark, where we have processed all sentinel-1 scan from since 2016.

Rapid robotic system development sets a demand for multi-disciplinary methods and tools to explore and compare design alternatives. In this paper, we present collaborative modeling that combines discrete-event models of controller software with continuous-time models of physical robot components. The presented co-modeling method utilized VDM for discrete-event and 20-sim for continuous-time modeling. The collaborative modeling method is illustrated with a concrete example of collaborative model development of a mobile robot animal feeding system. Simulations are used to evaluate the robot model output response in relation to operational demands. The result of the simulations provides the developers with an overview of the impacts of each solution instance in the chosen design space. Based on the solution overview the developers can select candidates that are deemed viable to be deployed and tested on an actual physical robot.

Mapping field environments into point clouds using a 3D LIDAR has the ability to become a new approach for online estimation of crop biomass in the field. The estimation of crop biomass in agriculture is expected to be closely correlated to canopy heights. The work presented in this paper contributes to the mapping and textual analysis of agricultural fields. Crop and environmental state information can be used to tailor treatments to the specific site. This paper presents the current results with our ground vehicle LiDAR mapping systems for broad acre crop fields. The proposed vehicle system and method facilitates LiDAR recordings in an experimental winter wheat field. LiDAR data are combined with data from Global Navigation Satellite System (GNSS) and Inertial Measurement Unit (IMU) sensors to conduct environment mapping for point clouds. The sensory data from the vehicle are recorded, mapped, and analyzed using the functionalities of the Robot Operating System (ROS) and the Point Cloud Library (PCL). In this experiment winter wheat (Triticum aestivum L.) in field plots, was mapped using 3D point clouds with a point density on the centimeter level. The purpose of the experiment was to create 3D LiDAR point-clouds of the field plots enabling canopy volume and textural analysis to discriminate different crop treatments. Estimated crop volumes ranging from 3500-6200 (m3) per hectare are correlated to the manually collected samples of cut biomass extracted from the experimental field.

A Light Detection and Ranging (LiDAR) sensor mounted on an Unmanned Aerial Vehicle (UAV) can map the overflown environment in point clouds. Mapped canopy heights allow for the estimation of crop biomass in agriculture. The work presented in this paper contributes to sensory UAV setup design for mapping and textual analysis of agricultural fields. LiDAR data are combined with data from Global Navigation Satellite System (GNSS) and Inertial Measurement Unit (IMU) sensors to conduct environment mapping for point clouds. The proposed method facilitates LiDAR recordings in an experimental winter wheat field. Crop height estimates ranging from 0.35–0.58 m are correlated to the applied nitrogen treatments of 0–300 kg N ha . The LiDAR point clouds are recorded, mapped, and analysed using the functionalities of the Robot Operating System (ROS) and the Point Cloud Library (PCL). Crop volume estimation is based on a voxel grid with a spatial resolution of 0.04 × 0.04 × 0.001 m. Two different flight patterns are evaluated at an altitude of 6 m to determine the impacts of the mapped LiDAR measurements on crop volume estimations.

The stricter legislation within the European Union for the regulation of herbicides that are prone to leaching causes a greater economic burden on the agricultural industry through taxation. Owing to the increased economic burden, research in reducing herbicide usage has been prompted. High-resolution images from digital cameras support the studying of plant characteristics. These images can also be utilized to analyze shape and texture characteristics for weed identification. Instead of detecting weed patches, weed density can be estimated at a sub-patch level, through which even the identification of a single plant is possible. The aim of this study is to adapt the monocot and dicot coverage ratio vision (MoDiCoVi) algorithm to estimate dicotyledon leaf cover, perform grid spraying in real time, and present initial results in terms of potential herbicide savings in maize. The authors designed and executed an automated, large-scale field trial supported by the Armadillo autonomous tool carrier robot. The field trial consisted of 299 maize plots. Half of the plots (parcels) were planned with additional seeded weeds; the other half were planned with naturally occurring weeds. The in-situ evaluation showed that, compared to conventional broadcast spraying, the proposed method can reduce herbicide usage by 65% without measurable loss in biological effect.

Rapid robotic system development has created a demand for multi-disciplinary methods and tools to explore and compare design alternatives. In this paper, we present a collaborative modeling technique that combines discrete-event models of controller software with continuous-time models of physical robot components. The proposed co-modeling method utilizes the Vienna development method (VDM) and MATLAB for discrete-event modeling and 20-sim for continuous-time modeling. The model-based development of a mobile robot mink feeding system is used to illustrate the collaborative modeling method. Simulations are used to evaluate the robot model output response in relation to operational demands. An example of a load-carrying challenge in relation to the feeding robot is presented, and a design space is defined with candidate solutions in both the mechanical and software domains. Simulation results are analyzed using design space exploration (DSE), which evaluates candidate solutions in relation to preselected optimization criteria. The result of the analysis provides developers with an overview of the impacts of each candidate solution in the chosen design space. Based on this overview of solution impacts, the developers can select viable candidates for deployment and testing with the actual robot.

Testing agricultural operations and management practices associated with different machinery, systems and planning approaches can be both costly and time-consuming. Computer simulations of such systems are used for development and testing; however, to gain the experience of real-world performance, an intermediate step between simulation and full-scale testing should be included. In this paper, a potential common framework using the LEGO Mindstorms NXT micro-tractor platform is described in terms of its hardware and software components. The performance of the platform is demonstrated and tested in terms of its capability of supporting decision making on infield operation planning. The proposed system represents the basic measures for developing a complete test platform for field operations, where route plans, mission plans, multiple-machinery cooperation strategies and machinery coordination can be executed and tested in the laboratory.