The Significance of CSI and Paper-Level Classification System in Coping with the Challenges Brought by Ultra-Highly Cited Papers to Journal Evaluation

Abstract The COVID-19 pandemic resulted in a surge in the production and high citation rates of related publications, and these ultra-highly cited papers brought grave challenges to journal evaluation. So it is significant to test the performance of bibliometric indicators during the crisis and assess their ability to adapt to rapidly evolving research landscapes. The CAS Journal Ranking, one of the most widely used journal ranking systems in China, is committed to accurately revealing the average impact of journals and enhancing the robustness of evaluation results. This study focused on the response of the CAS Journal Ranking system to the ultra-highly cited papers related to COVID-19. We compared the journal impact factor (JIF), category normalized citation impact (CNCI), and CAS’s indicator - the field normalized citation success index (FNCSI) - under journal-level and paper-level classification systems by assessing changes in indicator values and examining ranking mobility of journals. The results indicate combining FNCSI and CWTS paper-level classification system yields a robust indicator in coping with the challenges brought by COVID-19 papers. The combination is effective because FNCSI measure reduces the enormous impact of COVID-19 papers, while CWTS paper-level classification system groups the majority of COVID-19 papers into the “coronavirus” category, preventing distortion of citation normalization of other groups. By revealing the pros and cons of various indicators, we hope to emphasize the relative suitability and dependence on the context. and inform future improvements to scientific journal evaluation systems and methodologies.