Trimodality Fusion for Soft Gripper's Grasping Prediction.

Soft gripper's grasping prediction is a crucial task in industrial automation, which requires feedback from multiple modalities, such as vision and touch. However, multimodal fusion is challenging due to mismatched data dimensions and density. Meanwhile, the soft gripper's bending as a result of grasping force can be monitored by flex sensors, which may also be helpful for grasping prediction. Here we create a dataset of a fin ray effect soft gripper grasping various objects while being monitored by a camera, a pressure sensor, and a flex sensor. Next, we fuse the multimodal sensory data based on a deep learning model to predict the grasping outcome. The ablation experiment results show that the trimodality fusion model can effectively improve the soft gripper's grasping prediction performance.