Use of a Rubric for Pediatric Emergency Medicine Fellowship Application Review

Background There is a need to review a large number of applications for pediatric emergency medicine fellowship in a holistic and systemic, unbiased manner. There exists a need to restructure the application process. We sought to develop and implement a rubric screening rubric for initial evaluation of pediatric emergency medicine fellowship applications that avoided traditionally used metrics that may be biased against racially underrepresented groups who are historically excluded from medicine. Methods An interactive process was used by key program leadership with review of prior literature and input from Diversity, Equity, and Inclusivity departmental chair to develop a holistic screening rubric with consensus reached around key factors that aligned with our fellowship program mission. All applications were reviewed with the rubric by the program director or the associate program director. A subset of applications being considered for review were additionally scored by members of the fellowship selection committee. Results Numerical scores ranged from 2 to 14, with the maximum potential score being 14. Seventy percent of those applicants invited for interview scored 9 or higher. Reliability of scores between the program director and the associate program director was high (intraclass coefficient, 0.89); however, reliability between the program director or associate program director and the selection committee members was low to moderate (intraclass coefficient, 0.46). Conclusions Developmental and use of a rubric screening allowed our institution to reflect on our priorities, as well as avoid potential bias. The use of the tool allowed us to communicate about applications in an objective and consistent manner. As we continue to iterate on the rubric, we hope to incorporate additional criteria to better identify highly qualified applicants who may otherwise be overlooked in a traditional screening process and gain familiarity in reviewers use.