Spatial extent-aware multimodal fusion method for measuring urban socioeconomic status

The automatic measurement of socioeconomic status (SES), such as household income, provides fundamental data for policymakers and business applications. Although several urban data sources were employed in previous studies, the spatial extent of the measured objects was ignored, which left room for further improvement in the accuracy of measuring SES. This study develops a multimodal semantic segmentation framework to fuse the spatial extent of ground objects, remote sensing, and near-sensing images for predicting SES levels. The framework first constructs ground feature layer (GFL) tiles by projecting ground-level features from sparse ground images. Then, the ground feature tiles aggregate regional ground-level features with the help of the spatial extents of ranges. Lastly, an improved deep semantic segmentation network is employed to fuse GFL and remote sensing images to predict SES levels. Experimental results on the London dataset show that the proposed method outperforms SOTA models and is robust. The framework provides a rewarding exploration in fusing image data and spatial vector data for image-based intelligent applications and can be applied to a series of socioeconomic applications.