Quality of multiple-choice questions used in medical internship qualification examination at Debre Tabor University, Ethiopia

Abstract Background: Assessment of cognitive competence is a major element of the internship qualification exam in undergraduate medical education in Ethiopia. Assessing the quality of exam items can help to improve the validity of assessments and assure stakeholders about the accuracy of the go/no decision to the internship. However, we know little about the quality of exam items utilized to ascertain fitness to join the medical internship. Therefore, this study aimed to analyze the quality of multiple-choice questions (MCQs) of the qualification exam administered to final-year medical students at DTU, Ethiopia. Methods: A mixed-method cross-sectional study design was employed to assess the psychometric qualities of 120 randomly selected MCQs and 407 distractors. Item characteristics were estimated using the item response theory (IRT) model. T-test, one-way ANOVA, and chi-square tests were run to analyze the univariate association between factors. Spearman correlation test was done to determine the predictive validity of the qualification examination. Result: Overall, 16%, 51%, and 33% of the items had high, moderate, and low distractor efficiency, respectively. About two-thirds (65.8%) of the items had two or more functioning distractors and 42.5% exhibited a desirable difficulty index. However, 77.8% of items administered in the qualification examination had a negative or poor discrimination index. Four and five option items didn’t show significant differences in psychometric qualities. The qualification exam showed a positive predictive value of success in the national licensing examination (Pearson’s correlation coefficient = 0.5).Conclusions: The difficulty index, functionality of distractors, and item reliability were acceptable, while the discrimination index was poor. Five option items didn’t have better psychometric qualities than the four options. Student performance in the qualification examination had a positive predictive validity. Items used for high-stake examinations need to be designed carefully and reviewed before administration.