MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning
CoRR(2024)
摘要
Learning robust and scalable visual representations from massive multi-view
video data remains a challenge in computer vision and autonomous driving.
Existing pre-training methods either rely on expensive supervised learning with
3D annotations, limiting the scalability, or focus on single-frame or monocular
inputs, neglecting the temporal information. We propose MIM4D, a novel
pre-training paradigm based on dual masked image modeling (MIM). MIM4D
leverages both spatial and temporal relations by training on masked multi-view
video inputs. It constructs pseudo-3D features using continuous scene flow and
projects them onto 2D plane for supervision. To address the lack of dense 3D
supervision, MIM4D reconstruct pixels by employing 3D volumetric differentiable
rendering to learn geometric representations. We demonstrate that MIM4D
achieves state-of-the-art performance on the nuScenes dataset for visual
representation learning in autonomous driving. It significantly improves
existing methods on multiple downstream tasks, including BEV segmentation (8.7
IoU), 3D object detection (3.5
work offers a new choice for learning representation at scale in autonomous
driving. Code and models are released at https://github.com/hustvl/MIM4D
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要