LIGHT: Learning Image-text Grounding for Hierarchical Tumor Localization and Subtype Classification in Multimodal Brain MRI

Accurate MRI-based brain tumor analysis requires not only tumor subtype classification but also localization at an anatomical granularity that is consistent with radiology reports. Most vision-only methods address localization and classification as separate label-prediction tasks, and therefore provide limited alignment with the fine-grained anatomical semantics used in routine reporting. To address this limitation, we propose LIGHT (Learning Image-text Grounding for Hierarchical Tumor analysis), a 3D vision-language framework that formulates brain tumor localization and subtype classification as image-text retrieval in a shared embedding space. Given multimodal MRI inputs, LIGHT retrieves coarse anatomical regions, fine-grained subregions, and tumor subtypes in a unified coarse-to-fine procedure. The framework has three main components: (1) a hierarchical retrieval vocabulary containing 21 coarse-grained regions, 563 fine-grained subregions, and 5 tumor subtypes; (2) large-scale foundation pretraining on a curated patient-disjoint in-house dataset of 99,813 MRI-report pairs; and (3) grounded task fine-tuning that uses segmentation-guided tumor crops and multi-template prompt supervision to improve tumor-prompt alignment. Across 11,034 annotated 3D brain tumor MRI cases from in-house, external clinical, and public datasets, LIGHT achieved 72.1% accuracy for coarse-grained localization, 70.1% top-1 accuracy for fine-grained localization, and 76.0%, 83.7%, and 84.6% accuracy for subtype classification on three 3D datasets, respectively. Additional public 2D subtype-classification experiments provide further evidence of cross-setting subtype recognition. These results suggest that report-grounded image-text retrieval can provide interpretable anatomical and diagnostic outputs for brain tumor MRI analysis. Code: https://github.com/qiuzhaoyu/LIGHT.