Applying Inter-Rater Reliability to Improve Consistency in Classifying PhD Career Outcomes

In the past year, there has been an exciting groundswell of national efforts to integrate multiple taxonomies for the transparent dissemination and analysis of PhD career outcomes. In this study, we leveraged the unique resources of the Broadening Experiences in Scientific Training Consortium to examine the reliability of the three-tiered Unified Career Outcomes Taxonomy (UCOT v.2017) that was collaboratively developed at a meeting convened by Rescuing Biomedical Research in August 2017. Using an amended version of the UCOT v.2017 (UCOT v.2017-rev1) and a new Supplementary Guidance document, we categorized over 570 PhD alumni records from three different universities. Utilizing Krippendorff’s alpha to measure the interrater reliability from nine different individuals, we determined moderate to robust reproducibility within the first two tiers of the taxonomy (Workforce Sector and Career Type); however, the reliability for the third tier (Job Function) did not meet established standards. The team identified significant sources of error, revised category definitions, improved coder training materials and processes, and tested for improved reliability through coding 219 PhD alumni records using the revised taxonomy, UCOT v.2017-rev2. Our results revealed that the changes introduced in UCOT v.2017-rev2 improved inter-rater reliability in all three tiers, and either met or exceeded the acceptable standards for reliability. A final set of clarifications were made to UCOT v.2017-rev2, resulting in UCOT v.2018 and a Finalized Guidance document. Our findings underscore the importance of carefully developing guidance documents to aid coders in the reliable and consistent categorization of alumni career outcomes. We propose periodic assessment of the UCOT v.2018 to address the natural evolution of PhD careers in the global workforce. Ultimately, we hope that UCOT v.2018 will aid in the classification and dissemination of alumni career outcomes that is essential to educating trainees, institutions, and agencies about the diversity of career options for PhDs, and therein empower all PhDs to pursue the careers of their choice.