LLM4ST: Leveraging Large Language Models for Multimodal Spatiotemporal Modeling

Spatiotemporal data modeling plays a critical role in a wide range of real-world applications, including traffic forecasting, climate analysis, and urban sensing. While recent advances in transformers and graph neural networks have improved the ability to capture complex spatial and temporal dependencies, existing models often lack semantic understanding and contextual awareness, limiting the accuracy of spatiotemporal feature representation. In this work, we propose LLM4ST, a novel multimodal framework that leveraging large language models (LLMs) to enhance spatiotemporal representation learning. LLM4ST unifies three complementary modalities: (1) tokenized raw spatiotemporal data, (2) natural language descriptions that encode domain knowledge or contextual information, and (3) latent embeddings from spatiotemporal encoding backbones. These heterogeneous inputs are integrated via a transformer-based cross-modal encoder, enabling joint reasoning over numerical patterns and semantic cues. By incorporating descriptive language and token information as an auxiliary modality, LLM4ST introduces semantic knowledge priors, thereby enhancing the model’s generalization. Extensive experiments on real-world benchmarks demonstrate that LLM4ST outperforms strong baselines across traffic prediction. Moreover, ablation studies validate the complementary nature of language and data-driven features. Our work highlights the potential of LLMs as general-purpose priors for spatiotemporal understanding, bridging the gap between symbolic semantics and spatiotemporal representations. The codes are at https://github.com/CodingMaker2/LLM4ST.