Academic Influence Prediction

Paper impact Prediction. Given the metadata of the paper (including the title of the paper, the author, the published conference or journal, year, etc.) and the citation relationship between the papers, predict the influence of the paper 𝛥𝑦𝑟 years later, in order to quantify the influence, use the conference or a journal's "Test of Time Paper Award" serves as an indicator of the high impact of a paper.

Author Impact Prediction. Given the collection of papers from 𝑦𝑟 and 𝑦𝑟 years ago and the citation relationship between papers, each paper contains attributes such as title, author, published conference or journal, year, etc. The goal is to predict the author's citations in 𝑦𝑟 + 𝛥𝑦𝑟 year.

OAG-Paper-TOT: This dataset collates Test-of-Time paper award data for selected conferences and journals in computer science. The meaning given to a paper by the Test-of-Time Award is: the paper has produced a huge theoretical or applied influence several years after its publication. Awards with similar meanings include the Most Influential Award, Hall of Fame Award, etc. At present, a total of 1063 papers in the computer science field that are awarded by 2022 have been collected. The data set is divided according to the publication year of each conference or journal, resulting in a training set of 476 papers, a validation set of 202 papers, and a test set of 385 papers.

AuthPred-2017: This dataset contains a subset of AMiner authors and papers published by these authors before 2011, and the goal is to predict the citations of these authors as of 2016. This dataset uses AMiner's citation statistics. This dataset provides real data of 1 112 931 authors for training and 123 823 authors for testing.

AuthPred-2022: This dataset contains a subset of AMiner's authors in the field of computer science and the papers published by these authors before 2017. The goal is to predict the citations of these authors as of early 2022 (according to Google Scholar statistics). The authors are randomly divided, and the training set includes 16078 authors, the validation set includes 5359 authors, and the test set includes 5360 authors.

For the paper impact prediction, the prediction is made in the unit of the paper collection of each conference or journal. The true label of each paper in the paper collection is 0 or 1. 1 means that the paper has won the time Test-of-Time award, otherwise it is marked as 0. Then compare the predicted probability of winning the Test-of-Time award for each paper with the real label, and calculate the Mean Average Precision (MAP). The average of the MAP values of different conferences or journals is used as the evaluation metric.

For author impact prediction, the root mean square error of the predicted citation value and the actual citation value is used (Root Mean Squared Error, RMSE) as the evaluation metric.

The methods for paper impact prediction include:

Citation: This method only uses the total number of citations of the known years of the paper to predict, and the sigmoid function can be used to normalize the number of citations to [0, 1].

RandomForest (RF): This method uses the number of citations per year and the total number of citations as the feature, the papers are classified by random forest classifier. Positive examples select papers that have been awarded the Test-of-Time Award, and negative examples do the opposite. Due to the imbalance of positive and negative examples, we randomly sample negative examples and keep the ratio of positive and negative examples at 1:10.

GBRT: This method uses the same features and data processing step as RF, and uses gradient boosting decision trees as classifiers.

PageRank: This method uses the paper citation network to calculate the PageRank value of each paper, and the larger the PageRank value, the more influential it is considered.

GraphSAGE: This method regards the paper influence prediction as a semi-supervised node classification problem on the citation graph, and uses the GraphSAGE model to aggregate the representation of the paper from the neighbor nodes, and the aggregated representation is used for the binary classification of the paper nodes. Initial representations of paper nodes are computed using OAG-BERT encoded paper titles.

The methods for author impact prediction include:

ARIMA: ARIMA is a class of statistical models used for time series forecasting. ARIMA is an acronym that stands for Autoregressive Integrated Moving Average Model. It is an extension of the autoregressive moving average model.

Linear Regression: This method defines a series of characteristics of the author, including the author's annual citations in the past 20 years, the total number of citations, the total number of papers, the H index, and the author citations in the test set estimated by using the author citations in the training sets and paper-author relations, and other features, using a linear regression model to predict author citations.

GBRT: This method uses the same features as linear regression, and uses gradient boosting regression trees for regression prediction.

LSTM: This method uses the characteristics of the author in the past 20 years as a time series. The characteristics of each moment include the number of papers and citations of the author in that year, and LSTM is used for regression prediction.

EvolveGCN: This method models author influence prediction as a node regression problem on a dynamic graph. Firstly, the co-author network of different years is constructed, and the graph convolutional network (GCN) is used to model the co-author network. EvolveGCN uses RNN to update the parameters of GCN, and the goal is to predict the number of citations of the author node.

Code & Data:

https://github.com/THUDM/citation-prediction

https://github.com/THUDM/tot-prediction

We kindly request that any published research that makes use of this data cites the following papers.  

@inproceedings{zhang2024oag,
  title={OAG-bench: a human-curated benchmark for academic graph mining},
  author={Fanjin Zhang and Shijie Shi and Yifan Zhu and Bo Chen and Yukuo Cen and Jifan Yu and Yelin Chen and Lulu Wang and Qingfei Zhao and Yuqing Cheng and Tianyi Han and Yuwei An and Dan Zhang and Weng Lam Tam and Kun Cao and Yunhe Pang and Xinyu Guan and Huihui Yuan and Jian Song and Xiaoyan Li and Yuxiao Dong and Jie Tang},
  booktitle={Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  pages={6214--6225},
  year={2024}
}