Chrome Extension
WeChat Mini Program
Use on ChatGLM

Data-Free Low-Bit Quantization via Dynamic Multi-teacher Knowledge Distillation

PATTERN RECOGNITION AND COMPUTER VISION, PRCV 2023, PT VIII(2024)

Cited 0|Views33
No score
Abstract
Data-free quantization is an effective way to compress deep neural networks under the situation where training data is unavailable, due to data privacy and security issues. Although Off-the-shelf data-free quantization methods achieve the relatively same accuracy as the fully-precision (FP) models for high-bit (e.g., 8-bit) quantization, low-bit quantization performance drops significantly to restrict their extensive applications. In this paper, we propose a novel data-free low-bit quantization method via Dynamic Multi-teacher Knowledge Distillation (DMKD) to improve the performance of low-bit quantization models. In particular, we first introduce a generator to synthesize the training data based on the input of random noise. The low-bit quantization models are then trained on these synthetic images by the dynamic knowledge from the FP model and the high-bit quantization models, which are balanced by learnable loss weight factors. The factors are controlled by a tiny learnable FP network to adaptively allocate the balanced weights for the knowledge from the FP model and the high-bit quantization models during training. For inference, we only kept the low-bit quantization model by safely removing other additional networks, such as the generator and the tiny model. Extensive experiments demonstrate the effectiveness of DMKD for low-bit quantization of widely-used convolutional neural networks (CNNs) on different benchmark datasets. Our DMKD ooon methods.
More
Translated text
Key words
Data-Free Quantization,Knowledge Distillation,Dynamic knowledge,Loss Weight Factors
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined