Instance Segmentation and Localization of Strawberries in Farm Conditions for Automatic Fruit Harvesting

Accurate detection and localization of fruits is essential for strawberry harvesting robots. However, segmentation of strawberries in clusters and determination of ripeness remain challenging. Also, occlusions can result in inaccurate localization of fruits. This paper presents a method for detection, instance segmentation and better localization of strawberries, based on a deep convolutional neural network (DCNN). Four classes, including three for different ripeness levels of strawberries and one for deformed strawberries, were defined in the DCNN model. Results show that ripe strawberries are the easiest to be identified among the four classes. A bounding box refinement method was then proposed to improve the localization accuracy by detecting occluded fruits and recovering the actual fruit sizes using bounding boxes. The width to height ratio (WHR) of output masks was used to detect occlusions, and a corresponding refinement method based on the solidity of the mask shape was proposed to find the occluded side of the fruit. The refinement of occluded side is the final step, where we used the mean WHR of unoccluded strawberries to compensate the occluded part. The refinement method was assessed on the strawberry variety of ‘Lusa’, which shows it can estimate and recover the actual sizes. Comparison experiment shows that the bounding box overlap between the refined and ground truth is 0.87, while the overlap between raw detected and ground truth is 0.68. The result indicates that the refinement method can locate fruits more accurately.