Memory Consolidation Enables Long-Context Video Understanding
CoRR(2024)
摘要
Most transformer-based video encoders are limited to short temporal contexts
due to their quadratic complexity. While various attempts have been made to
extend this context, this has often come at the cost of both conceptual and
computational complexity. We propose to instead re-purpose existing pre-trained
video transformers by simply fine-tuning them to attend to memories derived
non-parametrically from past activations. By leveraging redundancy reduction,
our memory-consolidated vision transformer (MC-ViT) effortlessly extends its
context far into the past and exhibits excellent scaling behavior when learning
from longer videos. In doing so, MC-ViT sets a new state-of-the-art in
long-context video understanding on EgoSchema, Perception Test, and Diving48,
outperforming methods that benefit from orders of magnitude more parameters.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要