Knowledge Amalgamation Via Images in the Wild

The proliferation of pre-trained models has significantly influenced the landscape of deep learning, providing a solid foundation for solving various downstream tasks. Recently, Knowledge amalgamation (KA) has emerged as a powerful framework for reusing pre-trained models by training a multi-talented student model. However, a key challenge, in this context, arises from that many pre-trained models are released without accompanying training datasets. In this paper, we address this challenge by introducing a novel paradigm called Out-of-Domain Knowledge Amalgamation (OOD-KA), which leverages solely out-of-domain data in the wild to learn the student model from pre-trained models. At the core of our proposed OOD-KA approach lies the Patch Amalgamation method, which enables the generation of task-specific data and features by capturing and reassembling local patterns in the OOD data. These amalgamated features are then utilized to effectively merge the knowledge from pre-trained models into the student model. We validate our method on image classification and semantic segmentation tasks across various benchmark datasets, and the experimental results demonstrate that, in the absence of original data, OOD-KA still offers an effective approach for customizing versatile student models based on pre-existing models. Our code is available at https://github.com/WuDiHJQ/OOD-KA.