Research of Weibo Text Classification Based on Knowledge Distillation and Joint Model

Text classification is a basic task in natural language processing. In 2018, the BERT(Bidirectional Encoder Representation from Transformers) was proposed. This model greatly improves the effect of natural language processing related tasks. However, the model size of the pre-trained language model is large, and because of its huge network structure, the predict time is long. In order to solve these problems, an improved model that uses knowledge distillation and adversarial perturbation is proposed. In the training phase, RoBERTa-wwm-ext is used as the teacher model, the joint model of Text-CNN and Text-RCNN is used as a student model, combined with label smoothing and adversarial perturbation method to improve the classification accuracy of the student model, compared with Text-CNN on two datasets, is improved by 1.91% and 1.21% respectively. Using the student model to classify texts has the advantages of easy for deployment and taking less prediction time while obtaining ideal accuracy.