Vision-based size distribution analysis of rock fragments using multi-modal deep learning and interactive annotation

Real -time size distribution analysis is of great significance to rock fragments in practical engineering. Traditional methods have struggled to strike a balance between analysis speed and precision, prompting the recent adoption of deep learning. However, prevalent approaches for estimating rock fragment sizes from RGB images (singlemodality) suffer from two defects: (a)time-consuming and labour -intensive dataset annotation, (b) poor transferability between cases. To solve the above problems, a comprehensive multi -modal framework for size distribution prediction of rock fragments (SDPRF) is proposed in this paper. This framework comprises three essential components: a multi -modal image dataset generation method, a multi -modal rock surface net (Mrsnet) for fragment edge detection and a 2-step breakpoints connection algorithm. The test results indicate:(1) The generation method of SDPRF dataset greatly reduces the time required for dataset annotation, (2) Mrsnet shows better generalization ability for other cases outside the training set than traditional single -modal learning.