Class-Incremental Learning (CIL) aims to learn new knowledge without forgetting the old knowledge. One of the popular approaches is to obtain transferable representations, which would be general for learning incremental tasks without expanding the representations. Recently, many works focus on making the final representation more transferable across incremental tasks. However, researchers rarely focus on shallow layer representations and utilize their properties to facilitate CIL, although they are shown to be more transferable than the final representation. In this paper, we investigate the properties of the shallow layer representations and utilize them to improve the performance in class-incremental learning. Specifically, we show that shallow layer representations forget less than deeper layers. Furthermore, we find that shallow layer representations have more stable intra-class relations. Such intra-class relations reflect the task-agnostic information that the deeper layer representations lack. Therefore, we propose Intra-class Backward Distillation (IncBD) to make the deeper layers learn from the intra-class relations of the shallow layer's representations, making the final representation more stable in terms of the intra-class relations. To compensate for the loss of class separability introduced by backward distillation, we also propose to train auxiliary classifiers for each layer's representation. Extensive experiments are performed to show that the intra-class relations are important for the transferability of the final representation and performance improvement in class-incremental learning.
分享
引用
