Image restoration seeks to reconstruct high-quality images from low-quality observations, with tasks such as superresolution, deblurring, and inpainting. Recently, diffusion models have emerged as powerful tools for image restoration, generating diverse and high-quality samples. However, their high computational demands, including multiple denoising steps, hinder deployment on resource-constrained devices like smartphones. While efforts to optimize sampling trajectories have been made, the large parameters of DMs remain a challenge. Model quantization, a common solution for reducing latency and memory usage, faces channel-wise variance when applied to DMs. Thus, an incoherence processing is needed for the outlier suppression to produce approximately Gaussian distribution. Futhermore, the uniform bitwidth allocation also turns into the bottleneck of quantization due to the ignorance of layer-wise sensitivity. To address the two main challenges, we propose MPRDiff, a mixed-precision quantization method that uses a computationally efficient randomized Hadamard transform technique to eliminate channel-wise incoherency and enable W3.8A4 layer-wise quantization of the DMs. Additionally, we use integer linear programming for optimal mixed bitwidth allocation and incorporate sampling step correction techniques. MPRDiff is evaluated on CelebA and ImageNet, demonstrating a minimal increase in FID while significantly reducing model size.
原文链接
分享
引用
