UGPT: Uncertainty-Guided Dynamic Prompt Tuning for Vision-Language Models

Adapting frozen vision-language models (VLMs) to downstream tasks via prompt tuning has emerged as a parameter-efficient alternative to full fine-tuning. However, existing prompt tuning methods still lack explicit control over how strongly prompts should adapt to different samples, despite the significant variation in sample difficulty and prediction reliability-a critical limitation for trustworthy deployment in complex real-world scenarios such as autonomous perception and intelligent control systems. We propose UGPT (Uncertainty-Guided Dynamic Prompt Tuning), a framework that explicitly leverages visual prediction uncertainty as a sample-level control signal for prompt modulation. UGPT computes normalized entropy from a frozen zero-shot CLIP branch, maps it through a lightweight Uncertainty Injection Network (UIN) into prompt-space perturbation vectors, and applies broadcast injection to generate sample-adaptive dynamic prompts. Under the 16-shot few-shot setting, UGPT uses only 41 K trainable parameters in total, achieves 81.58% Top1 accuracy on ImageNet-1K, and obtains consistent improvements across four out-of-distribution benchmarks (Avg. OOD: 62.0%). Ablation studies and interpretability analyses confirm that the perturbation magnitude increases monotonically with uncertainty, validating the principled “harder samples receive stronger modulation“ mechanism.