Data-centric AI has recently proven to be more effective and high-performance, while traditional model-centric AI delivers fewer and fewer benefits. It emphasizes improving the quality of datasets to achieve better model performance. This field has significant potential because of its great practicability and getting more and more attention. However, we have not seen significant research progress in this field, especially in NLP. We propose DataCLUE, which is the first Data-Centric benchmark applied in NLP field. We also provide three simple but effective baselines to foster research in this field (improve Macro-F1 up to 5.7 comprehensive experiments with human annotators and show the hardness of DataCLUE. We also try an advanced method: the forgetting informed bootstrapping label correction method. All the resources related to DataCLUE, including datasets, toolkit, leaderboard, and baselines, is available online at https://github.com/CLUEbenchmark/DataCLUE

Pretrained Language Models (PLMs) have achieved tremendous success in natural language understanding tasks. While different learning schemes -- fine-tuning, zero-shot, and few-shot learning -- have been widely explored and compared for languages such as English, there is comparatively little work in Chinese to fairly and comprehensively evaluate and compare these methods and thus hinders cumulative progress. In this paper, we introduce the Chinese Few-shot Learning Evaluation Benchmark (FewCLUE), the first comprehensive few-shot evaluation benchmark in Chinese. It includes nine tasks, ranging from single-sentence and sentence-pair classification tasks to machine reading comprehension tasks. We systematically evaluate five state-of-the-art (SOTA) few-shot learning methods (including PET, ADAPET, LM-BFF, P-tuning and EFL), and compare their performance with fine-tuning and zero-shot learning schemes on the newly constructed FewCLUE benchmark. Experimental results reveal that: 1) The effect of different few-shot learning methods is sensitive to the pre-trained model to which the methods are applied; 2) PET and P-tuning achieve the best overall performance with RoBERTa and ERNIE respectively. Our benchmark is used in the few-shot learning contest of NLPCC 2021. In addition, we provide a user-friendly toolkit, as well as an online leaderboard to help facilitate further progress on Chinese few-shot learning. We provide a baseline performance on different learning methods, a reference for future research.

In the paper, we report the results for the NLPCC2021 shared-task of Few-shot Learning for Chinese NLP. This shared task is proposed in the context of pre-trained language models, where models only have access to limited human-labeled data. The goal of the task is to compare different learning schemes. The task includes nine sub-tasks and three task forms: single sentence classification, sentence pair classification, and machine reading comprehension. In order to accommodate the properties of few-shot learning, we sampled the examples using various sampling methods, some with 32 examples in total for one dataset, while others with 4 to 16 examples per class. Ninety teams registered for the shared task, employing a wide range of learning schemes, including data augmentation, utilizing multiple templates rather than a single template, using unlabeled data for pre-training or semi-supervised training. The best model achieved 65.3 in the mean accuracy, compared with the human score of 83.9. This result is 8 points higher than our baseline model (using the PET scheme). We believe our few-shot learning tasks and results demonstrate the potential of the recently introduced few-shot learning methods and provide guidance and important empirical evidence for future research.