ZIP Your Data: Length-Adaptive Visual Token Optimization for Efficient Multi-Modal Training

Recent advancements in multi-modal LLMs (MLLMs) have introduced length-adaptive visual token representations for flexible visual grounding. However, this adaptability presents new challenges for efficient training. Token compression, a representative acceleration method originally designed for fixed-length inputs, reintroducing padding and computational overhead when applied to length-adaptive visual tokens. To address this, we propose an optimized training framework Zip that seamlessly integrates multi-modal sample packing with visual compression. Specifically, our packing technique efficiently concatenates multiple length-adaptive multi-modal samples into shared instances, effectively eliminating padding overhead, and the progressive-staged compression method adaptively reduces visual token lengths across Transformer layers with minimal performance degradation. Experiments on LLaVA-NeXT showcase that our approach cuts Supervised Fine-Tuning (SFT) training time by up to 45%, while maintaining or even improving accuracy across 11 multi-modal benchmarks. These results highlight the practicality of the packing-coordinated compression strategy in fully max-imizing both training efficiency and model performance of length-adaptive token representations in MLLMs. Code is available at 1.