Vision Transformer (ViT) has achieved unprecedented success in vision tasks with the assistance of abundant data. However, the lack of inductive bias in lightweight ViT makes learning locality challenging on small datasets, leading to poor performance. This limitation impedes the application of lightweight ViT in scenarios with limited datasets and computational power. Knowledge Distillation (KD) allows student models to benefit from the teacher model. However, in the progressive learning stage, traditional single-stage KD methods are usually suboptimal for delivering fixed knowledge to the growing student model. To address these issues, we propose a simple yet effective two-stage KD method called Curriculum Information Knowledge Distillation (CIKD) for the first time. Specifically, we incorporate a curriculum learning framework, progressing from easy to difficult, in the KD curriculum. At the first stage, i.e., Attention Locality Imitation (ALI), the student model learns locality from the low-level semantic features of the teacher model through self-attention distillation. Afterward, at the second stage, i.e., Logit Mimicking (LM), the student model learns label information and high-level semantic logit from the teacher model. Without bells and whistles, our approach achieves state-of-the-art results on 8 small-scale datasets with ViT-Tiny (5.0M). Our code and model weights are available at: https://github.com/newLLing/CIKD.

Although humans are capable of learning new tasks without forgetting previous ones, most neural networks fail to do so because learning new tasks could override the knowledge acquired from previous data. In this work, we alleviate this issue by proposing a novel Efficient Perturbation Inference and Expandable Network (EPIE-Net), which dynamically expands lightweight task-specific decoders for new classes and utilizes a mixed-label uncertainty strategy to improve the robustness. Moreover, we calculate the average probability of perturbed samples at inference, which can generally improve the performance of the model. Experimental results show that our method consistently outperforms other methods with fewer parameters in class incremental learning benchmarks. For example, on the CIFAR-100 10 steps setup, our method achieves an average accuracy of 76.33% and the last accuracy of 65.93% within only 3.46M average parameters.

In this paper, our goal is to design a simple learning paradigm for long-tail visual recognition, which not only improves the robustness of the feature extractor but also alleviates the bias of the classifier towards head classes while reducing the training skills and overhead. We propose an efficient one-stage training strategy for long-tailed visual recognition called Global and Local Mixture Consistency cumulative learning (GLMC). Our core ideas are twofold: (1) a global and local mixture consistency loss improves the robustness of the feature extractor. Specifically, we generate two augmented batches by the global MixUp and local CutMix from the same batch data, respectively, and then use cosine similarity to minimize the difference. (2) A cumulative head-tail soft label reweighted loss mitigates the head class bias problem. We use empirical class frequencies to reweight the mixed label of the head-tail class for long-tailed data and then balance the conventional loss and the rebalanced loss with a coefficient accumulated by epochs. Our approach achieves state-of-the-art accuracy on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT datasets. Additional experiments on balanced ImageNet and CIFAR demonstrate that GLMC can significantly improve the generalization of backbones. Code is made publicly available at https://github.com/ynu-yangpeng/GLMC

Histopathological images, as the gold standard for cancer diagnosis, record abundant information about microscopic structures and morphological characteristics through staining and scanning. However, the stain variation caused by the deviations in the production process not only confuse the pathological diagnosis, but also depress the generalization of computer-aided diagnosis. Stain normalization provides an efficient preprocessing method, which generates images with the same stain style while preserving the texture information. In this paper, we extend the advantages of adversarial transfer learning and auto-encoder structure to propose an adversarial stain normalization network (ASNN). For the stain style alignment, ASNN liberalizes the selection for the specific template image and adopts a domain discriminator after encoding to eliminate the distribution discrepancies of latent features caused by stain variation. To ensure precisely stain style transfer, we reconstruct the target stain style image using a reconstruction loss-constrained autoencoder and by the loss of reconstruction so that the stain decoder learns the rules for texture-to-stain conversions. We simulate the experiments on two publicly available datasets and utilize multiple indicators to evaluate the qualities of generated images. By comparing classification performance on downstream tasks, ASNN exhibits its superiority than traditional stain normalization methods. Furthermore, ASNN adopts a lighter network structure, which is faster than traditional stain normalization methods, and can stain 500 images within 1 second.

Correctly identifying sleep stages is essential for assessing sleep quality and treating sleep disorders. However, the current sleep staging methods have the following problems: (1) Manual or semi-automatic extraction of features requires professional knowledge, which is time-consuming and laborious. (2) Due to the similarity of stage features, it is necessary to strengthen the learning of features. (3) Acquisition of a variety of data has high requirements on equipment. Therefore, this paper proposes a novel feature relearning method for automatic sleep staging based on single-channel electroencephalography (EEG) to solve these three problems. Specifically, we design a bottom–up and top–down network and use the attention mechanism to learn EEG information fully. The cascading step with an imbalanced strategy is used to further improve the overall classification performance and realize automatic sleep classification. The experimental results on the public dataset Sleep-EDF show that the proposed method is advanced. The results show that the proposed method outperforms the state-of-the-art methods. The code and supplementary materials are available at GitHub: https://github.com/raintyj/A-novel-feature-relearning-method .

Software is rapidly evolving and operates in a changing environment; therefore, in addition to software design and testing, it is essential to observe and understand the software execution behavior by modeling data recorded during the execution of the software to improve its reliability. The nested call relationship between methods during the execution of software is common, but most process-mining methods are unable to discover them, only generating flat models with low fitness. Meanwhile, it is easy to generate "spaghetti-like" models with low comprehensibility when dealing with complex software execution data. This paper proposes a component-based hierarchical software behavior model discovery method that can discover hierarchical nested call structures during software runtime, improving the fitness of the model; meanwhile, the proposed method partitions the discovery model into several parts by component information to improve the comprehensibility of the model, which can also reflect the interaction behavior within and between components. The proposed approach was implemented in a process mining toolkit. Using real-life software event logs and public datasets, we demonstrated that compared with other advanced process mining techniques, our approach can visualize actual software execution behavior in a more accurate and easy-to-understand way while balancing time performance.