Augmenting inter-rater concordance of radiologic extranodal extension in HPV-positive oropharyngeal carcinoma: A multicenter study

Objectives To assess intra- and inter-institutional concordance and identify methods to increase precision in radiologic extranodal extension (rENE) ascertainment in HPV+ oropharyngeal carcinoma. Methods Six radiologists, blinded to clinical outcomes, from three centers assessed rENE in two phases: Phase-I (20 cases) utilized each individual's a priori appreciation of the literature. Phase-II (30 additional cases) was performed after deliberating experience and consolidating operating definitions. Intra- and inter-institutional Kappa were calculated at >50% and >75% certainty levels, respectively. Results The Phase-I intra-institutional kappa was 0.76, 0.32, and 0.44 at >50% certainty and improved to 0.89, 0.61, and 0.66 at >75% certainty. Inter-institutional Fleiss' kappa also improved with higher certainty (from 0.40 to 0.57, p = 0.039). The Phase-II inter-rater kappa was significantly higher than Phase-I at the same certainty level (both p < 0.001). Conclusion A learning curve exists for rENE assessment. Strategies to augment reliability include high certainty for declaration, consolidated operating definitions, and sharing experience among radiologists.