Accreditation processes for health care professions are designed to ensure that individuals and programs in these fields meet established standards of quality and effectiveness. The accelerating pace of globalization in the health care professions has increased the need for a shared understanding of the vocabulary of evaluation, assessment, and accreditation. The psychometric principles of valid and reliable assessment are commonly accepted, but the terminology is confusing. We believe that all stakeholders – evaluators, faculty, students but also the community – will benefit from a shared language and common set of definitions. We recognize that not all readers will agree with the definitions we propose, but we hope that this guide will help to ensure clarity, consistency, transparency, and fairness, and that it will promote through the stimulation of a debate greater collaboration across national and international boundaries.

This is a step-by-step description of the content of the pre-sessions which was focused on clarifying concepts, explaining the case-based learning sessions sequences.

The outbreak of the COVID-19 pandemic was extraordinarily disruptive and presented medical educators with unprecedented challenges. The School of Medicine at University of Minho rapidly moved most curriculum activities to online formats, including our assessment processes. Our experience in remote assessment was very limited when we made the decision, and we knew that there was not enough time for extensive pilot testing. We were confident that we could leverage our overall expertise in assessment design and delivery, and we were able to gather valuable information access to and confidence with web-based instruments through student surveys. Recognising the various critical dimensions for valid remote assessment, the School of Medicine at University of Minho adopted the following parameters for remote assessment: Online examinations will be formative assessments. These examinations are designed to help students measure individual progress, and allow faculty to monitor the effectives of a remotely delivered curriculum and assessment process. There will be adequate student training for the assessment process and prompt feedback on performance. Item characteristics. Our examinations emphasise integration and problem solving rather than recall. Items are timed to minimise reliance on external sources. Test items are presented in random order to further protect security, and we trust students to act with integrity. Fairness. Fairness is a key aspect in assessment. As we cannot assure uniformity of testing conditions for remote examinations, and we do not know yet the impact of variable testing circumstances on exam performance, we decided to normalise scores to the average score of the past 3 cohorts in the same examination. This is based on the premise that each cohort of students is quite similar in terms of average performance. Student performance will be calculated as a z-score around average performance of the remote assessment examination and then normalised to a reference score of the past 3 years. This assures that students will not be disadvantaged when compared to previous and future cohorts. Freedom of choice. Because some students may feel uncomfortable with such format or disagree with the normalising procedure, students will be given the freedom to request a priori, assessment through oral examination without penalty or prejudice. Transparency and emotional support. Students contributed to the process that resulted in the current assessment model and will be continually consulted throughout delivery. This assures transparency and also significantly decreases emotional distress in a very complex context. We consider that this strategy represents good practice in remote assessment and may be transferable to other contexts and learning environments. So far, the evidence gathered indicates that it helped students gain trust in the assessment process and assures that assessment will achieve its main purpose as a critical tool to improve learning.

Musculoskeletal disorders and injuries account for a large percentage of primary care and emergency room visits in the United States. There is concern about potential shortages of orthopaedic specialists and an expectation that some portion of musculoskeletal care will be delivered by nonphysician health care providers. Physician assistants are one such group. Grunfeld and his co-investigators express their belief that all health care providers should have a basic fund of musculoskeletal knowledge that permits them to provide appropriate, safe, and efficient care. This is reasonable, and, in fact, the public largely believes that this is so. Unfortunately, there are few cross-disciplinary studies of the knowledge base of health care providers. The authors of this multicenter study used a valid and highly reliable standardized assessment of musculoskeletal knowledge produced by the National Board of Medical Examiners (NBME) to test the hypothesis that the basal knowledge of musculoskeletal medicine is equivalent in medical students and physician assistant students nearing graduation. Acknowledging that knowledge is but one competency required for practice, Grunfeld et al. found a significant difference. The mean examination score (and standard deviation) for medical students was higher than that for physician assistant students (73.8% ± 9.7% compared with 62.3% ± 11%; p < 0.05). Among medical students, those with more hours of exposure to specific areas covered by the examination performed better in those areas than students who lacked additional experience. Medical students interested in a career in musculoskeletal medicine or surgery performed better than those with a different interest. If examinations are well prepared, students of equivalent ability who have more training and higher levels of interest score higher. The findings of this study support the validity statements for the examination. Overall, the medical students in this study performed about as well in musculoskeletal medicine as North American medical students do in other areas of basic and clinical science covered by NBME examinations. An internal NBME study of performance on Step-3 musculoskeletal content conducted at the start of the Bone and Joint Decade demonstrated very similar findings. This is good news for faculty concerned with musculoskeletal education. Physician assistant students performed less well on this examination. The performance difference is real, but lesser knowledge is only one of the potential explanations. Physician assistant students have fewer overall years of medical education and less exposure to United States Medical Licensing Examination (USMLE) review materials and may be less able than medical students in multiple choice examinations. One should resist the temptation to make too much of this finding. Differential performance on multiple-choice examinations reflects different levels of knowledge, not overall competency. Clinical and communication skills, judgment, and professional behavior are equally important. Undergraduate examinations such as the USMLE Step-1 and Step-2 examinations and the NBME discipline-specific subject examinations are designed to measure an individual’s readiness to enter supervised postgraduate training, not to begin unsupervised practice. The medical students destined for specialization in orthopaedics, physical medicine, and rheumatology who participated in this experiment will complete five or more years of postgraduate training before they enter practice. During that period of time, their knowledge, technical skills, clinical judgment, and decision-making skills will grow with experience. Physician assistants may begin practice immediately on certification and licensure, but under supervision. The requirements for supervision vary among jurisdictions from direct oversight to availability by telephone or radio contact if necessary. One would hope that the level of responsibility accorded to physician assistants is appropriate to training and experience, but further study will be necessary to determine if, in similar circumstances, patient outcomes are the same for physicians and physician assistants. In conclusion, competent clinical care combines knowledge, skill, judgment, and communication. Knowledge is necessary but insufficient alone. Although medical students had higher scores than physician assistant students on the NBME Musculoskeletal Subject Examination, it would be inappropriate to conclude from this study that one cohort or the other would be better prepared to provide clinical musculoskeletal care.

Click to increase image sizeClick to decrease image size Acknowledgments This work was completed while the first author was at the National Board of Medical Examiners. We wish to thank the NBME for their permission to publish this research.

Background: Though progress tests have been used for several decades in various medical education settings, a few studies have offered analytic frameworks that could be used by practitioners to model growth of knowledge as a function of curricular and other variables of interest.Aim: To explore the use of one form of progress testing in clinical education by modeling growth of knowledge in various disciplines as well as by assessing the impact of recent training (core rotation order) on performance using hierarchical linear modeling (HLM) and analysis of variance (ANOVA) frameworks.Methods: This study included performances across four test administrations occurring between July 2006 and July 2007 for 130 students from a US medical school who graduated in 2008. Measures-nested-in-examinees HLM growth curve analyses were run to estimate clinical science knowledge growth over time and repeated measures ANOVAs were run to assess the effect of recent training on performance.Results: Core rotation order was related to growth rates for total and pediatrics scores only. Additionally, scores were higher in a given discipline if training had occurred immediately prior to the test administration.Conclusions: This study provides a useful progress testing framework for assessing medical students’ growth of knowledge across their clinical science education and the related impact of training.

BACKGROUND:To gather evidence of external validity for the Foundations of Medicine (FOM) examination by assessing the relationship between its subscores and local grades for a sample of Portuguese medical students.METHOD:Correlations were computed between six FOM subscores and nine Minho University grades for a sample of 90 medical students. A canonical correlation analysis was run between FOM and Minho measures.RESULTS:Moderate correlations were noted between FOM subscores and Minho grades, ranging from -0.02 to 0.53. One canonical correlation was statistically significant. The FOM variate accounted for 44% of variance in FOM subscores and 22% of variance in Minho end-of-year grades. The Minho canonical variate accounted for 34% of variance in Minho grades and 17% of the FOM subscore variances.CONCLUSIONS:The FOM examination seems to supplement local assessments by targeting constructs not currently measured. Therefore, it may contribute to a more comprehensive assessment of basic and clinical sciences knowledge.

Background: The Ministry of Health of the Republic of Panama is currently developing a national examination system that will be used to license graduates to practice medicine in that country, as well as to undertake postgraduate medical training. As part of these efforts, a preliminary project was undertaken between the National Board of Medical Examiners (NBME) and the Faculty of Medicine of the University of Panama to develop a Residency Selection Process Examination (RSPE).Purpose: The purpose of this study was to assess the reliability and validity of RSPE scores for a sample of candidates who wished to obtain a residency slot in Panama.Methods: The RSPE, composed of 200 basic and clinical sciences multiple-choice items, was administered to 261 residency applicants at the University of Panama.Results: The reliability estimate computed was comparable with that reported with other high-stakes examinations (Cronbach's alpha = 0.89). Also, a Rasch examinee proficiency item difficulty plot showed that the RSPE was well targeted to the proficiency levels of candidates. Finally, a moderate correlation was noted between local grade point averages and RSPE scores for University of Panama students (r = 0.38).Conclusions: Findings suggest that it is possible to translate and adapt test materials for use in other contexts. Copyright (C) 2005 by Lawrence Erlbaum Associates, Inc.

The introduction of a clinical skills examination (CSE) to Step 2 of the U.S. Medical Licensing Examination (USMLE) has focused attention on the design and delivery of large-scale standardized tests of clinical skills and raised the question of the appropriateness of evaluation of these competencies across the span of a physician's career. This initiative coincides with growing pressure to periodically assess the continued competence of physicians in practice. The USMLE CSE is designed to certify that candidates have the basic clinical skills required for the safe and effective practice of medicine in the supervised environment of postgraduate training. These include history taking, physical examination, effective communication with patients and other members of the health care team, and clear and accurate documentation of diagnostic impressions and plans for further assessment. The USMLE CSE does not assess procedural skills. As physicians progress through training and enter practice, both knowledge base and requisite technical skills become more diverse. A variety of indirect and direct measures are available for evaluating physicians, but, at present, no single method permits high-stake inferences about clinical skills. Systematic and standardized assessments make a contribution to comprehensive evaluations, but they retain an element of assessing capacity rather than authentic performance in practice. Much work is needed to identify the optimal combination of methods to be employed in support of programs to ensure maintenance of competence of practicing physicians.

Medical training is undergoing extensive revision in France. A nationwide comprehensive clinical competency examination will be administered for the first time in 2004, relying exclusively on essay-questions. Unfortunately, these questions have psychometric shortcomings, particularly their typically low reliability. High score reliability is mandatory in a high-stakes context. The National Board of Medical Examiners-designed multiple choice-questions (MCQ) are well adapted to assess clinical competency with a high reliability score. The purpose of this study was to test the hypothesis that French medical students could take an American-designed and French-adapted comprehensive clinical knowledge examination with this MCQ format. Two hundred and eighty five French students, from four Medical Schools across France, took an examination composed of 200 MCQs under standardized conditions. Their scores were compared with those of American students. This examination was found assess French students' clinical knowledge with a high level of reliability. French students' scores were slightly lower than those of American students, mostly due to a lack of familiarity with this particular item format, and a lower motivational level. Another study is being designed, with a larger group, to address some of the shortcomings of the initial study. If these preliminary results are replicated, the MCQ format might be a more defendable and sensible alternative to the proposed essay questions.

PURPOSE:The French government, as part of medical education reforms, has affirmed that an examination program for national residency selection will be implemented by 2004. The purpose of this study was to develop a French multiple-choice (MC) examination using the National Board of Medical Examiners' (NBME) expertise and materials.METHOD:The Evaluation Standardisée du Second Cycle (ESSC), a four-hour clinical sciences examination, was administered in January 2002 to 285 medical students at four university test sites in France. The ESSC had 200 translated and adapted MC items selected from the Comprehensive Clinical Sciences Examination (CCSE), an NBME subject test.RESULTS:Less than 10% of the ESSC items were rejected as inappropriate to French practice. Also, the distributions of ESSC item characteristics were similar to those reported with the CCSE. The ESSC also appeared to be very well targeted to examinees' proficiencies and yielded a reliability coefficient of.91. However, because of a higher word count, the ESSC did show evidence of speededness. Regarding overall performance, the mean proficiency estimate for French examinees was about 0.4 SD below that of a CCSE population.CONCLUSIONS:This study provides strong evidence for the usefulness of the model adopted in this first collaborative effort between the NBME and a consortium of French medical schools. Overall, the performance of French students was comparable to that of CCSE students, which was encouraging given the differences in motivation and the speeded nature of the French test. A second phase with the participation of larger numbers of French medical schools and students is being planned.

both know and do. In the practice of medicine, knowledge is applied in the context of patient encounters, and physicians must have the skills necessary to gather information from (and transmit information to) patients, families, and colleagues. Research studies and practical experience indicate that some physicians have an adequate knowledge base, but display unsatisfactory clinical and communication skills. Such physicians pose a potential threat to the public, at least as great as that posed by clinicians with inadequate knowledge.