Impact of reference panel composition on scores of script concordance test assessing basic nephrology knowledge in undergraduate medical education

PurposeIn the assessment of basic medical knowledge, the composition of the reference panel between specialists and primary care (PC) physicians is a contentious issue. We assessed the effect of panel composition on the scores of undergraduate medical students in a script concordance test (SCT).MethodsThe scale of an SCT on basic nephrology knowledge was set by a panel of nephrologists or a mixed panel of nephrologists and PC physicians. The results of the SCTs were compared with ANOVA for repeated measurements. Concordance was assessed with Bland and Altman plots.ResultsForty-five students completed the SCT. Their scores differed according to panel composition: 65.6 & PLUSMN; 9.73/100 points for nephrologists, and 70.27 & PLUSMN; 8.82 for the mixed panel, p < 0.001. Concordance between the scores was low with a bias of -4.27 & PLUSMN; 2.19 and a 95% limit of agreement of -8.96 to -0.38. Panel composition led to a change in the ranking of 71% of students (mean 3.6 & PLUSMN; 2.6 places).ConclusionThe composition of the reference panel, either specialist or mixed, for SCT assessment of basic knowledge has an impact on test results and student rankings.Practice pointsThe composition of the reference panel, either specialist or mixed (specialist and primary care (PC) physicians) for SCT assessment of basic knowledge has an impact on test results and student rankings.The choice of the reference panel to assess basic knowledge should be discussed according to educational objectives before testing.