Enabling Visually Aware Conversational Data Visualization Through LLM Augmentation.

We present an augmentation method that makes Large Language Models (LLMs) both data aware and visually aware. Unaugmented LLMs can provide high-quality information about the broad context of a visualization, but are unaware of the visual content and thus cannot provide accurate, visualization-specific answers. We address this limitation by providing LLMs with structured metadata generated from a combination of extracted visual information and textual descriptions. Our LLM-agnostic approach preprocesses the visualization to extract features from it using a vision model, combines them with textual information about the data, and generates a compact JSON file that then augments the LLM during user interactions. This highly structured file provides the LLM with the necessary multimodal context without requiring any fine-tuning or costly multimodal prompts, and applies to any existing prerendered visualization paired with descriptive text. We demonstrate our method using geospatial datasets from the Science On a Sphere project. A user study confirms our system's accuracy and appeal to users.