Do deep neural networks utilize the weight space efficiently?
CoRR(2024)
摘要
Deep learning models like Transformers and Convolutional Neural Networks
(CNNs) have revolutionized various domains, but their parameter-intensive
nature hampers deployment in resource-constrained settings. In this paper, we
introduce a novel concept utilizes column space and row space of weight
matrices, which allows for a substantial reduction in model parameters without
compromising performance. Leveraging this paradigm, we achieve
parameter-efficient deep learning models.. Our approach applies to both
Bottleneck and Attention layers, effectively halving the parameters while
incurring only minor performance degradation. Extensive experiments conducted
on the ImageNet dataset with ViT and ResNet50 demonstrate the effectiveness of
our method, showcasing competitive performance when compared to traditional
models. This approach not only addresses the pressing demand for parameter
efficient deep learning solutions but also holds great promise for practical
deployment in real-world scenarios.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要