Selective Visual Program Reasoning in Whole-Slide Large Vision-Language Models

Large vision-language models (VLMs) show potential in pathology image diagnosis but struggle with fine-grained feature extraction from gigapixel whole-slide images (WSIs). To address this, we propose a two-stage framework that uses external visual programs to enhance visual input for pathology question answering. First, an LLM-guided visual program selection module trains the VLM to choose relevant programs and generate interpretable rationales. Second, the selected programs are executed on WSIs to extract detailed visual features, which are then fused into the VLM for accurate slide-level QA. We also introduce SlideVLM, a vision-language model backbone with a visual token selection mechanism designed for efficient WSI processing in both stages. Experiments show significant accuracy improvements on the WSI-level visual question answering task. Project link: https://github.com/Lewislou/SVPR.