COMLEX-USA and USMLE for Osteopathic Medical Students: Should We Duplicate, Divide, or Unify?

In 2020, approximately 60% of osteopathic medical students took at least one portion of the United States Medical Licensing Examination (USMLE).1 At first, this may seem a curious statistic. After all, the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) is the only examination accepted for osteopathic physician licensure in every jurisdiction in the United States,2 and DO-granting medical schools must require (as a condition of their accreditation) their graduates to pass the first 2 portions of the COMLEX-USA.3 Since both COMLEX-USA and USMLE have validity evidence as licensing examinations,4 this duplication is wasteful: while all states require successful completion of one examination, no state requires 2.Instead, most osteopathic students who take the USMLE do so to gain advantage in residency selection. Data from the National Resident Matching Program demonstrate that DO students who report taking the USMLE match into their preferred specialties at higher rates than their peers who take only the COMLEX-USA.5 Although these groups differ in ways beyond their choice of licensing examinations, differences in match rates likely fuel the perception that DO applicants should take both tests. For instance, among DO senior applicants in internal medicine, 96.3% successfully matched after taking USMLE Step 1 compared to 88.7% who reported taking only COMLEX-USA Level 1; in emergency medicine, 86.6% matched vs 75.8%; and in general surgery, 74.9% matched vs 49.0%.5Whatever osteopathic medical students gain by taking the USMLE comes at a price. The examination itself is expensive, both in its literal and opportunity costs. For this reason, the National Board of Osteopathic Medical Examiners (NBOME) has published percentile calculators,6 and other groups have published conversion equations7–10 in the hope that these might allow program directors to more easily assess relative performance on COMLEX-USA.In this issue of the Journal of Graduate Medical Education, Barnum et al, on behalf of the NBOME, present the most authoritative concordance study of COMLEX-USA and USMLE scores to date.11 Their study was large, with 2115 DO students who took both examinations between 2015 and 2020, and includes students from 5 colleges of osteopathic medicine, thus providing more generalizable results than previous single center studies. Unsurprisingly, there was substantial correlation between COMLEX-USA and USMLE scores, but in contrast to previous analyses that have used linear regression, Barnum et al utilized equipercentile matching to better evaluate what seems in fact to be a curvilinear relationship. The centerpiece of their work is a table of COMLEX-USA to USMLE conversions, which the NBOME intends to develop into a tool that could be used by residency program directors when evaluating DO applicants.Though the study is methodologically sound, the inference that can be drawn regarding an individual examinee's likely USMLE score based upon their performance on COMLEX-USA is still constrained by inherent data limitations. First, DO students do not take the USMLE at random: students who choose to take both tests tend to score more highly on COMLEX-USA than those who do not (by 25 points/0.28 SD for Level 1 or 40 points/0.43 SD for Level 2-CE in the present study). Selection bias may therefore reduce the accuracy of projected scores when extrapolated to the entire population of osteopathic medical students. Second, and more importantly, the correlation between COMLEX-USA and USMLE performance—though consistent—is imprecise. Examination of the scatterplots shows observed USMLE scores that extend well above and below the projected value from equipercentile matching. For examinees scoring at the mean for the study population—560 for COMLEX-USA Level 1 and 600 for the Level 2-CE—observed USMLE scores vary across an approximately 50-point range.Still, in a world in which there is a single accreditation system for graduate medical education, it is inefficient—if not inexcusable—that DO medical students should be compelled to take 2 licensure examinations to be fairly considered for residency training positions. Even with their limitations, the concordance tables presented by Barnum et al offer a viable solution to this problem. However, it is not the only one.An alternative was recently proposed by the Coalition for Physician Accountability's Undergraduate Medical Education to Graduate Medical Education Review Committee, which recommended that program directors receive only a single normalized score for an applicant's licensing examination, regardless of whether that examination was the USMLE or COMLEX-USA.11 Because this normalized score could be used as a filterable variable in the Electronic Residency Application Service (ERAS), such an approach would offer program directors labor-saving advantages over the use of concordance tables, percentile calculators, or predictive equations.Implementing this recommendation would require buy-in from ERAS and allopathic physicians, who could justifiably be concerned that such a system would disadvantage MD applicants. While DO students can take either examination, MD students are not permitted to take COMLEX-USA. Yet among students who take both COMLEX-USA and the USMLE, test-takers tend to have better relative performance on the former than the latter. For example, while Barnum et al report concordance between a COMLEX Level 1 score of 640–659 and a USMLE Step 1 score of 240–244,11 such an applicant's COMLEX-USA score would typically fall around the 90th percentile,5 while their USMLE score would fall around the 67th.12 Similarly, while there may be concordance between a COMLEX-USA Level 2-CE score of 600–619 and a USMLE Step 2 CK score of 238–241, the former corresponds to a percentile score around the 70th percentile; the latter, the 35th.6,13 While such differences could be due to the lack of osteopathic content on the USMLE, there are systematic differences between DO and MD matriculants on the Medical College Admission Test.14,15Alternatively, the NBOME could—without the assent of any other organization—simply report the results of the COMLEX-USA using a 300-point scale, similar to the scaled score reporting of the USMLE. The choice of any scaled score is arbitrary and mutable: until 1991, the National Board of Medical Examiners (NBME) used an 800-point range for its Part I and Part II examinations.16 A common scale would allow program directors (or the automated filters they deploy in ERAS) to immediately compare relative performance between examinees but might still raise concerns related to systematic differences between the examinees of each test as noted above.Yet each of these potential solutions ignores what could be the most fair and logical solution: a single licensure examination for all physicians. The USMLE was created in part to address the perceived inequity of the dual system of examination previously in place for US MD and international medical graduates by creating a common standard by which physicians could be evaluated, regardless of their medical school location.17 Notably, the NBOME initially expressed interest in participating in the unified effort to create a single licensing examination before choosing instead to develop the COMLEX-USA.18A single licensing examination has broad support among the future of the medical profession. In spring 2020, the Student Osteopathic Medical Association, the student arm of the American Osteopathic Association, passed a resolution calling upon the NBOME, NBME, and Federal State Medical Boards to create a single licensing examination and for the Commission on Osteopathic College Accreditation to remove COMLEX-USA as a graduation requirement for DO medical students, and to instead require a new single licensing examination.19 The American Medical Association Medical Student Section passed the same resolution in 2021.20In fact, by demonstrating the substantial concordance between USMLE and COMLEX-USA scores, Barnum et al may provide additional evidence that either examination could reasonably stand in for the other. Examination of the scatterplot of COMLEX-USA and USMLE scores shows significant agreement between the 2 examinations, including in the failing range, where inference is most important. This builds upon previous research from the NBOME, which found that only 0.8% of test takers passed USMLE but failed COMLEX-USA.21When considering the possibility of a single licensure examination, it must be acknowledged that COMLEX-USA and USMLE test somewhat different subject material, and only the former evaluates osteopathic principles. Yet whether these principles can be tested as a condition of licensure only in a 3-part examination series, or whether they might reasonably be assessed elsewhere, remains up for debate. Only 6 states—California, Florida, Oklahoma, Michigan, Pennsylvania, and West Virginia—specifically require COMLEX-USA for initial DO licensure; all other state boards accept the USMLE as well.2Barnum et al have added substantially to the body of work on the relationship between COMLEX-USA and USMLE scores. They have demonstrated convincingly that there is substantial concordance between performance on COMLEX-USA Level 1/Level 2-CE and USMLE Step 1/Step 2 CK, and they specified more precisely the relationship between these scores in a large and generalizable cohort of osteopathic medical students. All that remains unclear is what, exactly, the increasingly unified house of medicine should do with this knowledge. Should we continue to allow DO students to duplicate effort and expense by taking 2 similar examinations? Should we instead utilize workarounds to compare performance from one side of a divided system to the other? Or should MD and DO physicians come together to develop a unified standard to assess the capability of all physicians to provide safe and effective care?