LQER: Low-Rank Quantization Error Reconstruction for LLMs
CoRR(2024)
Abstract
Post-training quantization of Large Language Models (LLMs) is challenging. In
this work, we introduce Low-rank Quantization Error Reduction (LQER), which
combines quantization and low-rank approximation to recover the model
capability. LQER leverages an activation-induced scale matrix to drive the
singular value distribution of quantization error towards a desirable
distribution, which enables nearly-lossless W4A8 quantization on various LLMs
and downstream tasks without the need for knowledge distillation, grid search,
or gradient-base iterative optimization. Unlike existing methods, the
computation pattern of LQER eliminates the need for specialized Scatter and
Gather processes to collect high-precision weights from irregular memory
locations. Our W4A8 LLMs achieve near-lossless performance on six popular
downstream tasks, while using 1.36× fewer hardware resources than the
leading state-of-the-art method. We will open-source our framework once the
paper is accepted.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined