Recommending Peer Reviewers of Scientific Manuscripts from Massive Researchers

We study the problem of recommending the most appropriate reviewers for the peer review of scientific manuscripts from the massive researchers in the whole research community. It is different from the general expert finding problem, which mainly focuses on improving the associations between queries and experts but ignores some specific constraints; and it is also different from the reviewer assignment problem, in which the submissions and candidate reviewers are fixed.

We propose a novel and systematic reviewer recommendation framework, in which different search strategies, i.e., inverted indexing and graph-based search, are employed to find the relevant papers of the new manuscript. And during the search process, we respectively use three kinds of pairwise document similarity means, i.e., lexical similarity, semantic similarity and deep similarity, to measure the content relevance between papers. Finally we sum up the similarity scores of the relevant papers into their authors and recommend the most appropriate authors as reviewers by combining the expertise relevance, activity and diversity of the reviewers.

We conduct the experiments of recommending reviewers for some real conference papers in the field of computer science based on the large-scale academic dataset of the ArnetMiner system, and the experimental results demonstrate that our proposed framework can effectively recommend appropriate peer reviewers.

And we also implement an online reviewer recommender system, which is now on trial in recommending reviewers for several international journals in the field of computer science. It has been demonstrated that our system can significantly improve the efficiency of their reviewer invitation.

Method

We propose a systematic framework to recommend peer reviewers for scientific manuscripts from massive researchers in the whole research community. The overall framework of our reviewer recommender system is shown in the figure below. When a new scientific manuscript comes, we first search its most relevant (i.e., similar) papers in the entire paper set , then sum up the similarity scores of the relevant papers into their authors and finally recommend the most appropriate authors as reviewers by combining the expertise relevance, activity and diversity of the reviewers.

Three search strategies are employed to find the relevant papers of the new manuscript. The first one is inverted indexing (II), the most common term-based search method without considering the citation relationships between papers. While the other two are more sophisticated and performed in the context of citation networks, which are path-based breadth-first search (BFS) and network-based random walk with restarts (RWR). Of course, we can further combine inverted indexing with one of the two citation-based strategies for better searching.

During the search process, whichever search strategy is used, one key problem is how to measure the content relevance between documents, i.e., the pairwise document similarity. In inverted indexing, we need to calculate the relevance between the new manuscript and the searched papers; while in citation-based search, we need to use the pairwise document similarities as the weights of citation relations. Three kinds of pairwise document similarity measures, i.e., lexical similarity, semantic similarity and deep similarity are introduced in our reviewer recommendation framework.

When summing up the similarity scores to authors, we also consider the effect of publishing years of papers. We assume that the weights of papers decrease with the increasing of their ages. Finally, the expertise relevance, activity and diversity of authors are simultaneously taken into account to recommend the final appropriate peer reviewers.

Datasets

We perform our experiments based on the large-scale academic dataset of the ArnetMiner system (http://aminer.org/), a free online service for academic social network analysis and mining. From the system we collect more than 2,000,000 papers and 1,700,000 authors in the field of computer science and form a citation network containing more than 8,000,000 edges, a paper2author network containing more than 5,100,000 edges and a co-author network containing more than 4,200,000 edges. This basic dataset is available at http://aminer.org/AMinerNetwork.

Then we collect 1131 papers accepted by 7 international conferences in 2013 as our benchmark dataset. The 7 conferences cover different research directions in the field of computer science. The title, authors, abstract and references of each paper are completely collected. In addition, the program committee (PC) members of each conference are also collected for the evaluation of our proposed framework. The statistics of our benchmark dataset are summarized in the table below. [Benchmark Dataset Download]

Experimental Results

In our reviewer recommendation framework, we combine the three kinds of pairwise document similarity measures with the five kinds of search strategies, and get 15 combination methods as shown in the table below. In our experiments we compare the performance of the 15 different combination methods.

We use the measure of precision at top K to qualify the reviewer recommendation performance. The following figure gives the performance of the 15 combination methods individually on each conference and the average performance on all the conferences. The heights of the bars (from the bottom) with different colors in the figures respectively represent the precisions at different K values.

From the experimental results we can see: 1) Our proposed reviewer recommendation framework can effectively recommend appropriate peer reviewers for scientific manuscripts in different research directions in the field of computer science; 2) The performances of semantic similarity and deep similarity are a little better than that of lexical similarity, while the difference between semantic similarity and deep similarity is not obvious at all; 3) The citation-based search (i.e., BFS or RWR) is much better than the term-base search (i.e., inverted indexing), which demonstrates that the citation relationships between papers can significantly improve the performance of searching relevant papers of the new manuscript; 4) By combining the term-based inverted indexing with one of the citation-based search methods (either BFS or RWR), we can further significantly improve the recommendation performance, which demonstrates the complementarity between the two kinds of search strategies.

Online System

We implement an online system of our proposed reviewer recommendation framework in the field of computer science based on the large-scale academic dataset of the ArnetMiner system (http://aminer.org).

Users input the title, authors, abstract, keywords and references of a new manuscript, and the system recommend at most the most appropriate 200 reviewers from the whole computer science research community. Users can filter the reviewer results by changing the scales of paper number, citation number, h-index and activity of the reviewers, and also they can limit or exclude the reviewers of some specific affiliations. In addition, for visually presenting the diversity between the recommended reviewers, the system further provides the co-author map of the recommended reviewers.

The address of our online system is: http://reviewer.aminer.org

Our system is now under the trials of recommending peer reviewers for several international journals, including SCIENCE CHINA Information Sciences (http://www.springer.com/computer/journal/11432), Journal of Computer Science and Technology (http://www.springer.com/computer/journal/11390) and Frontiers of Computer Science (http://www.springer.com/computer/journal/11704).

Reference

Huaiyu Wan, Junfeng Li, Renyi Cai, Jie Tang. Recommending Peer Reviewers of Scientific Manuscripts from Massive Researchers. Submitted to JASIST.