Cross-task feature enhancement strategy in multi-task learning for harvesting Sichuan pepper

It is challenge to accurately perceive multi-objective functions in an unstructured, dynamic, and undefined agricultural environment. Owing to the neglect of multi-task relationships, some existing multi-task networks (MTN) for agricultural perception lack enough feature representation ability to perform robust prediction in an unstructured environment. To assist automatic harvesting of the Sichuan pepper, we develop a perception model and improve it by propose a cross-task feature enhancement (CTFE) strategy to exploit the multi-task relationships. CTFE comprises a feature harmonization module (FHM) and a series of asymmetrical feature sharing and fusing module (AFSFM). The FHM is devised to reconstruct the semantic feature from the object detection decoder. To this end, the semantic feature was aligned with the segmentation features in terms of scale and channel, which enabled the segmentation decoder to exploit semantic information from the detect decoder to filter background noise and activate responses of the shielded area. Subsequently, the AFSFM was embedded into two segmentation decoders to capture the complementary information used for segmentation boundaries alignment. The visualization results indicate that AFSFM can assist to spread the semantic information. Finally, the improved perception model was used to perform three tasks: foreground pepper detection, pepper segmentation, and stem segmentation. Experiments on the Sichuan pepper dataset demonstrate that the proposed CTFE strategy significantly improves the multi-task performance by transferring semantic features to suppress segmentation noise and enhance feature representation. Moreover, the proposed strategy yields competitive performance in each task and can assist generic multi-task learning methods to further enhance multi-task performance.