MemQuant: Towards Memory-aware Post-Training Quantization for Multimodal Transformers

Recently Transformer model has revolutionized the fields of natural language processing (NLP) and computer vision (CV). Even though it shows unprecedented efficiency and accuracy on most tasks, memory and computation overhead can be huge, making it inadequate for real-world applications such as edge devices. Post-training quantization (PTQ) can be an efficient way to overcome this bottleneck. In this paper, we propose a novel memory-aware mixed precision PTQ which will consider memory consumption of blocks and sub-blocks inside transformer model. Our motivation originates from profiling result which varies according to application. We evaluate the proposed methodology by employing both for language model for translation tasks and a vision-to-language model for image captioning tasks. Through our approach, we achieve significant memory savings while experiencing small degradation in accuracy. We are able to achieve 10x memory savings for the weights of the annotated transformer, 5x for T5 text-to-text, and 5.5x for weights of vision-to-language model.