Reviewer Recommendation

Given a paper submission s_i ∈ S, the task goal is find actual reviewer(s) r_j ∈ R who are considered as suitable to review this submission. The submission and reviewer bases (i.e. S and R) are explicitly given, and basic information of each submission and reviewer is also provided.

Frontiers dataset

We generate a relatively hard but real-world dataset that recorded submissions published in Frontiers, an open-access publisher, which open-sourced the peer-review record. In this dataset, we collect 40,182 submissions associated with 42,587 reviewers.  Each submission has at least two corresponding reviewers. In this task, we present a training set, a validation set, and a test set. For each submission, we show at least one suitable reviewer(s) in the training set, and randomly keep the rest suitable reviewers in the validation or test set. The performance on the test set will be used as a final comparison. Additionally, the title, abstract, keywords, and author's information of each submission as well as the expertise and keywords of reviewers are also provided.

We use Recall@K, and NDCG@K for evaluation.

TF-IDF: encoding reviewers' keywords and keywords of submissions into semantic vectors, and calculating the similarity of each submission-reviewer pair.

GF-CF: proposes graph filter-based CF and proves the special cases via the lens of graph signal processing as well as the importance of smoothness.

LightGCN: simplifies the design GCN that abandons feature transformation and nonlinear activation, adopts the most essential component neighborhood aggregation for collaborative filtering, and uses a weighted sum of embeddings of all layers as its final embeddings.

Code & Data: https://github.com/THUDM/Reviewer-Rec

We kindly request that any published research that makes use of this data cites the following papers.  

@inproceedings{zhang2024oag,
  title={OAG-bench: a human-curated benchmark for academic graph mining},
  author={Fanjin Zhang and Shijie Shi and Yifan Zhu and Bo Chen and Yukuo Cen and Jifan Yu and Yelin Chen and Lulu Wang and Qingfei Zhao and Yuqing Cheng and Tianyi Han and Yuwei An and Dan Zhang and Weng Lam Tam and Kun Cao and Yunhe Pang and Xinyu Guan and Huihui Yuan and Jian Song and Xiaoyan Li and Yuxiao Dong and Jie Tang},
  booktitle={Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  pages={6214--6225},
  year={2024}
}