Toward Multimodal Vertical Federated Learning: A Traffic Analysis Case Study

Federated Learning (FL) is an emerging subclass of Artificial Intelligence that decentralizes the learning process. Unlike the well-studied Horizontal Federated Learning (HFL), which requires the feature space of all participants to be the same, the newly emerging Vertical Federated Learning (VFL) allows participants to hold different features, provided the sample space is the same. This unique aspect enables VFL to incorporate features from different data modalities, a capability that has not yet been sufficiently explored. Currently, VFL researchers adapt datasets originally used for HFL by splitting the data vertically, whether it is text, tabular, or image data. In this paper, we extend the application of VFL to multimodal datasets, specifically in the field of Intelligent Transportation. We build models by combining local models from participants holding CCTV image datasets and Traffic flow tabular datasets. Due to the absence of suitable existing datasets, we introduce a new dataset, the INDOT traffic dataset, which also supports sequential training across time and distance. Our experiments demonstrate the efficiency of VFL in the multimodal traffic analysis scenario and aim to expand the scope of VFL research.