Paper Source Tracing

Given a paper 𝑝 (including the full text of the paper) and its references, the goal is to find the most important references (called ref-source) from the references. ref-source largely inspired the paper 𝑝 in terms of ideas or methods. A paper can have one or more important references. For each reference in the paper 𝑝, an importance score in the range [0, 1] is calculated.

Among them, the importance of the reference refers to: (1) the main idea of the paper 𝑝 is inspired by the reference paper; (2) the main method of the paper 𝑝 comes from the reference paper. In other words, this paper would not be complete without these important references. Note that if paper 𝑝𝑐 cites paper 𝑝𝑎 and paper 𝑝𝑏, 𝑝𝑎 is an important reference for 𝑝𝑏 and 𝑝𝑏 is an important reference for 𝑝𝑐, then 𝑝𝑎 is not an important reference for 𝑝𝑐 (although 𝑝𝑐 cites 𝑝𝑐). That is, we only look for important references that directly inspired the paper 𝑝.

OAG-Paper-Trace: Since the labeling of paper source tracing requires strong expertise, dozens of graduate students in the computer science field were hired to let them mark the source of papers in their familiar fields. The data collection methods were divided into two groups. The first group asked each student to mark the papers they had read before, with an average of 20 papers per person. In order to continuously obtain high-quality labeled data, the second group expects each student to read two new papers every week in the form of a paper reading check-in, and to mark the source papers of the papers they read. After collection and preprocessing, the labeled data of 1120 papers in the computer field were obtained. According to the year when the papers were published, 560 papers are used for training, 280 papers are used for verification, and the remaining 280 papers are used for testing.

For each paper 𝑝 to be traced, the list of references is coded with 0-1 according to the labeling results (1 if it is ref-trace, 0 otherwise), and the prediction result of each reference is compared to the labeling result, calculating the Mean Average Precision (MAP) value. The MAP calculated on different papers is averaged as the evaluation metric.

Rule: This method analyzes the contexts of each reference. If words such as "motivated by" or "inspired by" appear in the context, the reference is considered to be an important reference, otherwise the reference is an ordinary reference.

ProNE: This method models the overall citation network of papers in the field of computer science (using DBLP citation network), and uses the ProNE algorithm to learn a vector representation for each paper. ProNE is an unsupervised network representation learning method. It first uses sparse matrix decomposition to learn the initial representation of nodes and then lets the node representations propagate in the spectral space after network modulation to obtain enhanced node representations. This method measures the importance of references to the target paper by calculating the cosine similarity between the paper representation and the reference representation.

BERT: This method concatenates the context of the references in the full text of the paper to obtain the text input, and then uses the BERT model to encode the text, and uses the reference annotation results in the training set as supervisory information for training. Considering the imbalance of the ratio of positive and negative examples, the negative examples are randomly sampled so that the ratio of positive and negative examples is 1:10.

Code & Data: https://github.com/THUDM/paper-source-trace

We kindly request that any published research that makes use of this data cites the following papers.

@article{zhang2024pst,
  title={PST-Bench: Tracing and Benchmarking the Source of Publications},
  author={Fanjin Zhang and Kun Cao and Yukuo Cen and Jifan Yu and Da Yin and Jie Tang},
  journal={arXiv preprint arXiv:2402.16009},
  year={2024}
}

@inproceedings{zhang2024oag,
  title={OAG-bench: a human-curated benchmark for academic graph mining},
  author={Fanjin Zhang and Shijie Shi and Yifan Zhu and Bo Chen and Yukuo Cen and Jifan Yu and Yelin Chen and Lulu Wang and Qingfei Zhao and Yuqing Cheng and Tianyi Han and Yuwei An and Dan Zhang and Weng Lam Tam and Kun Cao and Yunhe Pang and Xinyu Guan and Huihui Yuan and Jian Song and Xiaoyan Li and Yuxiao Dong and Jie Tang},
  booktitle={Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  pages={6214--6225},
  year={2024}
}