Scholar Profiling

Subtask 1: Search engine-based scholar profiling

Given a scholar's name, organization, and his/her search engine records (with query "name + organization"), the task goal is to predict or extract the scholar's attributes, such as homepage, gender, and position.

Subtask 2: Multi-dimensional scholar profiling based on long texts

Given a long text about a scholar, the goal is to extract different attributes of the scholar, including gender, position, organization, research interest, award, academic service, highest degree, honorary title, place of birth, date of birth, work experience, and education experience (12 attributes in total). The result of each attribute extraction marks the start position and end position in its text. Among all attributes, work experience and education experience are long attributes with lengths possibly greater than 512 words.

CCKS2021-En: This dataset comes from AMiner, which is an English subset of CCKS 2021 scholar profiling track. This dataset includes 9921 scholars. The dataset is split randomly, of which 5557 instances are used as training sets, 1833 instances are used for validation, and 1831 instances are used as test sets.

Scholar-XL: This dataset comes from the text description of the scholar's official homepage, and each attribute is manually marked with its starting and ending position in the text. This dataset includes 2099 scholars and the dataset is randomly split. 1400 instances are used as training sets, 349 instances are used for validation sets, and 350 instances are used as test sets.

For CCKS2021-En dataset, similar to CCKS2021 competition, we use the accuracy of the full match to compare the extraction results and ground truths. For Scholar-XL dataset, we adopt Precision, Recall, and F1 as evaluation metrics. We first evaluate the results of each attribute and then take a macro-average to calculate the final scores.

Following the winning solutions of CCKS 2021 and the SOTA named entity recognition (NER) methods, we select the following baselines.

SML: This method uses hand-crafted features for each attribute, and employs traditional classifiers.

- SML-esb: For gender prediction, we calculate the frequencies of signal words (e.g. "his", "her") at different positions and use decision trees, SVM, and logistic regression as classifiers. Majority voting is used for inference.

- Logistic Regression (LR): For homepage extraction, we extract some features, including the ranking positions of a URL in search results, whether some signal words (e.g. "edu", "academic") appear in the URL link, etc. Then we use LR as the classifier.

- XGBoost: For homepage extraction, we use the same features and use XGBoost as the classifier.

Rule: For position extraction, this method finds the location of the "position" with regular expression, and then selects the "position" appearing the most frequently as the prediction results.

BERT:

For gender prediction, we concatenate the scholar's name, organization, and webpage as input, and then fine-tune BERT for classification.

For homepage extraction, we concatenate the scholar's name and candidate URL as the input and then fine-tune BERT for classification.

For position extraction, we concatenate the scholar's name, organization, and web content with position information as the input, and then fine-tune BERT for classification.

BI-LSTM-CRF: This method uses BI-LSTM to model bi-directional features, and then uses CRF layers for sequence labeling. This method is used for position extraction.

BERT-prompt: This method uses pre-training models and prompt tuning for attribute extraction. The prompt template is defined as "{text} + is + [MASK]". The objective is text infilling. This method is used for gender prediction and position extraction.

For the Scholar-XL dataset, we adopt the following baselines.

CNN: This method uses a Biaffine decoder to generate the features for each starting position and ending position, and then employs CNN to model the spatial correlation for classification.

Global Pointer: This method employs pre-training models, and uses two feedforward layers to extract the features of starting position and ending position. In addition, this method uses a multiplication attention mechanism to integrate the relative position encoding for the starting position and ending position. This method also proposes an improved cross-entropy loss to relieve the class imbalance issue.

UIE: This method proposes a general-purpose generative information extraction framework. This method defines different structure extraction languages for different attributes and uses specific prompts to generate the results.

Code and Data: https://github.com/THUDM/scholar-profiling

English Scholar-XL Dataset:  https://github.com/THUDM/scholar-profiling/tree/main/bio_models/en_bio

Chinese Scholar-XL Dataset:  https://open-data-set.oss-cn-beijing.aliyuncs.com/oag-benchmark/scholar-profiling/zh_labeled.zip

We kindly request that any published research that makes use of this data cites the following papers.  

@inproceedings{zhang2024oag,
  title={OAG-bench: a human-curated benchmark for academic graph mining},
  author={Fanjin Zhang and Shijie Shi and Yifan Zhu and Bo Chen and Yukuo Cen and Jifan Yu and Yelin Chen and Lulu Wang and Qingfei Zhao and Yuqing Cheng and Tianyi Han and Yuwei An and Dan Zhang and Weng Lam Tam and Kun Cao and Yunhe Pang and Xinyu Guan and Huihui Yuan and Jian Song and Xiaoyan Li and Yuxiao Dong and Jie Tang},
  booktitle={Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  pages={6214--6225},
  year={2024}
}