Learning new concepts while preserving the learned knowledge is the goal for learning systems in Class-Incremental Learning (CIL). Previous methods mainly approach CIL from two perspectives: one focuses on preserving the learned knowledge, making the model compatible with previously learned tasks, i.e., backward-compatible; the other considers making the model prepared for learning new tasks, so the model should also be compatible with the future tasks, i.e., forward-compatible. In this paper, we systematically identify one of the fundamental challenges behind CIL, named feature collision, where the features learned by the current task-specific model may collide with those of the previous models, leading to forgetting of previously learned tasks and hindering the learning of new tasks. Therefore, we propose a bi-compatible CIL framework to promote continual learning by addressing feature collision in both forward and backward compatible manners. Specifically, we consider: 1) forward-compatible task-agnostic feature learning to capture task-agnostic features continually with a separate model in advance for subsequent tasks; 2) backward-compatible rehearsal feature augmentation to preserve the discrimination ability of the learned task-specific feature. The framework is called Bi-compatible Task-agnostic Feature Expansion and Augmentation (BitFea), comprising both forward-compatible feature expansion and backward-compatible feature augmentation. Firstly, it captures task-agnostic features continually with a separate model, providing extra task-agnostic features for subsequent tasks. Secondly, to obtain useful features from the task-agnostic model for the current task, it aggregates the task-agnostic features with the task-specific feature using a merge attention. Then the aggregated feature is transferred back into the task-specific feature for inference, helping the task-specific model capture diverse features. Lastly, the task-specific feature of the rehearsal samples is augmented by a gradient ascent step to help keeping the generalization on the previous tasks, preserving the discrimination ability of the task-specific feature across the learned tasks. Extensive experiments show the effectiveness and superiority of BitFea on various CIL settings. The code is available at https://github.com/bwnzheng/BitFea_IJCV .

The ability to learn new concepts while preserve the learned knowledge is desirable for learning systems in Class-Incremental Learning (CIL). Recently, feature expansion of the model become a prevalent solution for CIL, where the old features are fixed during the training of the new task while new features are expanded for the new tasks. However, such task-specific features learned from the new task may collide with the old features, leading to misclassification between tasks. Therefore, the expanded model is often encouraged to capture diverse features from the new task, aiming to avoid such collision. However, the existing solution is largely restricted to the samples from the current task, because of the poor accessibility to previous samples. To promote the learning and transferring of diverse features across tasks, we propose a framework called Task-Agnostic Guided Feature Expansion (TagFex). Firstly, it captures task-agnostic features continually with a separate model, providing extra task-agnostic features for subsequent tasks. Secondly, to obtain useful features from the task-agnostic model for the current task, it aggregates the task-agnostic features with the task-specific feature using a merge attention. Then the aggregated feature is transferred back into the task-specific feature for inference, helping the task-specific model capture diverse features. Extensive experiments show the effectiveness and superiority of TagFex on various CIL settings. Code is available at https://github.com/bwnzheng/TagFex_CVPR2025.

Class-Incremental Learning (CIL) seeks to learn new concepts without forgetting previously learned knowledge. To achieve this, rehearsal-based methods keep a replay memory consisting of a small number of trained samples from previous tasks. However, recent studies show that rehearsal-based methods are prone to overfitting on rehearsal samples, resulting in poor generalization on previous tasks. Since the generalization error is bounded by the margin on the training dataset, in this paper, we study the generalization by all-layer margin on deep neural networks to alleviate catastrophic forgetting. Specifically, we show that the average margin of the rehearsal samples are smaller during incremental learning. To acquire larger margin thus better generalization on rehearsal samples, we propose Multi-layer Rehearsal Feature Augmentation (MRFA) in rehearsal training to optimize the all-layer margin on rehearsal samples. The proposed method augments the features of rehearsal samples at each layer by gradient ascent step of the current model with respect to the feature. With such augmentations on layer features, the margin on rehearsal samples are larger, rehearsal samples are able to provide more information for refining the decision boundary during incremental learning, thus alleviating catastrophic forgetting. Extensive experiments show the effectiveness of MRFA on various CIL scenarios.

Class-Incremental Learning (CIL) aims to learn new knowledge without forgetting the old knowledge. One of the popular approaches is to obtain transferable representations, which would be general for learning incremental tasks without expanding the representations. Recently, many works focus on making the final representation more transferable across incremental tasks. However, researchers rarely focus on shallow layer representations and utilize their properties to facilitate CIL, although they are shown to be more transferable than the final representation. In this paper, we investigate the properties of the shallow layer representations and utilize them to improve the performance in class-incremental learning. Specifically, we show that shallow layer representations forget less than deeper layers. Furthermore, we find that shallow layer representations have more stable intra-class relations. Such intra-class relations reflect the task-agnostic information that the deeper layer representations lack. Therefore, we propose Intra-class Backward Distillation (IncBD) to make the deeper layers learn from the intra-class relations of the shallow layer's representations, making the final representation more stable in terms of the intra-class relations. To compensate for the loss of class separability introduced by backward distillation, we also propose to train auxiliary classifiers for each layer's representation. Extensive experiments are performed to show that the intra-class relations are important for the transferability of the final representation and performance improvement in class-incremental learning.

Learning new classes without forgetting is crucial for real-world applications for a classification model. Vision Transformers (ViT) recently achieve remarkable performance in Class Incremental Learning (CIL). Previous works mainly focus on block design and model expansion for ViTs. However, in this paper, we find that when the ViT is incrementally trained, the attention layers gradually lose concentration on local features. We call this interesting phenomenon as Locality Degradation in ViTs for CIL. Since the low-level local information is crucial to the transferability of the representation, it is beneficial to preserve the locality in attention layers. In this paper, we encourage the model to preserve more local information as the training procedure goes on and devise a Locality-Preserved Attention (LPA) layer to emphasize the importance of local features. Specifically, we incorporate the local information directly into the vanilla attention and control the initial gradients of the vanilla attention by weighting it with a small initial value. Extensive experiments show that the representations facilitated by LPA capture more low-level general information which is easier to transfer to follow-up tasks. The improved model gets consistently better performance on CIFAR100 and ImageNet100. The source code is available at https://github.com/bwnzheng/LPA_ICME2023.