Academic Question Answering

Given an academic question q, a set of N academic passages P^q = {p^q_1, p ^q_2,..,p^q_N} that is related to question q (N is usually a very large number, up to million), the task goal is to find an academic passage out of the candidate set P^q, which is the most possible to answer the given question. For each academic passage, a title and abstract are provided.

OAG-QA dataset

We construct an academic QA dataset from real questions from Question-and-Answers (Q&A) forums. We retrieve question posts from StackExchange and Zhihu and extract the paper URLs cited by users in their answers. The paper is aligned with OAG and Semantic Scholar. The dataset contains 17,948 question-paper pairs and we divide the question-paper pairs into 22 disciplines and 87 topics respectively and has a two-level hierarchical structure where each topic is under a specific discipline. Given each topic, we sample 10,000 candidate papers including the ground truth.

We use Hit@K for retrieval accuracy, which measures the percentage of questions that contain the gold paper in the top k retrieved papers. We report the Hit@20 for each topic and the average scores over each discipline.

DPR-FT: This method uses the Dense Retrieval DPR (DPR) model as the backbone. DPR adopts the standard BERT model and dual-encoder structure, and this method uses the OAG-QA dataset to fine-tune all training parameters of DPR.

DPR-PT2: This method also uses the DPR model as the backbone. Different from DPR-FT, this method uses P-Tuning v2 to efficiently fine-tune the parameters of the DPR model. P-Tuning v2 adds a small number of trainable parameters to each layer head of the model while keeping the model backbone parameters unchanged.

ColBERT-FT: This method uses the dense retrieval model ColBERT as the backbone. ColBERT combines twin-tower encoders and cross-encoders to encode questions and documents and employs a late interaction model to estimate the correlation between questions and documents. ColBERT adopts more fine-grained multi-vector representations for questions and documents. This method fine-tunes all the training parameters of ColBERT with the OAG-QA dataset.

ColBERT-PT2: This method also uses the ColBERT model as the backbone. Different from ColBERT-FT, this method uses P-Tuning v2 to efficiently fine-tune the parameters of the ColBERT model.

Code & Data: https://github.com/THUDM/P-tuning-v2/tree/main/PT-Retrieval

We kindly request that any published research that makes use of this data cites the following papers.  

@inproceedings{zhang2024oag,
  title={OAG-bench: a human-curated benchmark for academic graph mining},
  author={Fanjin Zhang and Shijie Shi and Yifan Zhu and Bo Chen and Yukuo Cen and Jifan Yu and Yelin Chen and Lulu Wang and Qingfei Zhao and Yuqing Cheng and Tianyi Han and Yuwei An and Dan Zhang and Weng Lam Tam and Kun Cao and Yunhe Pang and Xinyu Guan and Huihui Yuan and Jian Song and Xiaoyan Li and Yuxiao Dong and Jie Tang},
  booktitle={Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  pages={6214--6225},
  year={2024}
}

@inproceedings{tam2023parameter,
  title={Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers},
  author={Tam, Weng and Liu, Xiao and Ji, Kaixuan and Xue, Lilong and Liu, Jiahua and Li, Tao and Dong, Yuxiao and Tang, Jie},
  booktitle={Findings of the Association for Computational Linguistics: EMNLP 2023},
  pages={13117--13130},
  year={2023}
}