MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels
Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems(2024)
摘要
Human activity recognition (HAR) will be an essential function of various
emerging applications. However, HAR typically encounters challenges related to
modality limitations and label scarcity, leading to an application gap between
current solutions and real-world requirements. In this work, we propose MESEN,
a multimodal-empowered unimodal sensing framework, to utilize unlabeled
multimodal data available during the HAR model design phase for unimodal HAR
enhancement during the deployment phase. From a study on the impact of
supervised multimodal fusion on unimodal feature extraction, MESEN is designed
to feature a multi-task mechanism during the multimodal-aided pre-training
stage. With the proposed mechanism integrating cross-modal feature contrastive
learning and multimodal pseudo-classification aligning, MESEN exploits
unlabeled multimodal data to extract effective unimodal features for each
modality. Subsequently, MESEN can adapt to downstream unimodal HAR with only a
few labeled samples. Extensive experiments on eight public multimodal datasets
demonstrate that MESEN achieves significant performance improvements over
state-of-the-art baselines in enhancing unimodal HAR by exploiting multimodal
data.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要