CBQ: Cross-Block Quantization for Large Language Models
CoRR(2023)
摘要
Post-training quantization (PTQ) has played a key role in compressing large
language models (LLMs) with ultra-low costs. However, existing PTQ methods only
focus on handling the outliers within one layer or one block, which ignores the
dependency of blocks and leads to severe performance degradation in low-bit
settings. In this paper, we propose CBQ, a cross-block reconstruction-based PTQ
method for LLMs. CBQ employs a cross-block dependency using a homologous
reconstruction scheme, establishing long-range dependencies across multiple
blocks to minimize error accumulation. Furthermore, CBQ incorporates a
coarse-to-fine preprocessing (CFP) strategy for suppressing weight and
activation outliers, coupled with an adaptive LoRA-Rounding technique for
precise weight quantization. These innovations enable CBQ to not only handle
extreme outliers effectively but also improve overall quantization accuracy.
Extensive experiments show that CBQ achieves superior low-bit quantization
(W4A4, W4A8, W2A16) and outperforms existing state-of-the-art methods across
various LLMs and datasets. Notably, CBQ quantizes the 4-bit LLAMA1-65B model
within only 4.3 hours on a single GPU, achieving a commendable tradeoff between
performance and quantization efficiency.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要