Purpose Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. medical schools. The authors developed an online KFE to assess third-year medical students' decision-making abilities during internal medicine (IM) clerkships in the United States. They used Messick's unified validity framework to gather validity evidence regarding response process, internal structure, and relationship to other variables. Method From February 2012 through January 2013, 759 students (at eight U.S. medical schools) had 75 minutes to complete one of four KFE forms during their IM clerkship. They also completed a survey regarding their experiences. The authors performed item analyses and generalizability studies, comparing KFE scores with prior clinical experience and National Board of Medical Examiners Subject Examination (NBME-SE) scores. Results Five hundred fifteen (67.9%) students consented to participate. Across KFE forms, mean scores ranged from 54.6% to 60.3% (standard deviation 8.4-9.6%), and Phi-coefficients ranged from 0.36 to 0.52. Adding five cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to, respectively, 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (P < .01). Conclusions These results provide strong evidence for response-process and relationship-to-other-variables validity and moderate internal structure validity for using a KFE to complement other assessments in U.S. IM clerkships.

Background: Clinical reasoning is a complex skill students have to acquire during their education. For educators it is difficult to explain their reasoning to students, because it is partly an automatic and unconscious process. Virtual Patients (VPs) are used to support the acquisition of clinical reasoning skills in healthcare education. However, until now it remains unclear which features or settings of VPs optimally foster clinical reasoning. Therefore, our aims were to identify key concepts of the clinical reasoning process in a qualitative approach and draw conclusions on how each concept can be enhanced to advance the learning of clinical reasoning with virtual patients. Methods: We chose a grounded theory approach to identify key categories and concepts of learning clinical reasoning and develop a framework. Throughout this process, the emerging codes were discussed with a panel of interdisciplinary experts. In a second step we applied the framework to virtual patients. Results: Based on the data we identified the core category as the "multifactorial nature of learning clinical reasoning". This category is reflected in the following five main categories: Psychological Theories, Patient-centeredness, Context, Learner-centeredness, and Teaching/Assessment. Each category encompasses between four and six related concepts. Conclusions: With our approach we were able to elaborate how key categories and concepts of clinical reasoning can be applied to virtual patients. This includes aspects such as allowing learners to access a large number of VPs with adaptable levels of complexity and feedback or emphasizing dual processing, errors, and uncertainty.

Background: Considerable evidence in the learning sciences demonstrates the importance of engagement in online learning environments. The purpose of this work was to demonstrate feasibility and to develop and collect initial validity evidence for a computer-generated dynamic engagement score based on student interactions in an online learning environment, in this case virtual patients used for clinical education. Methods: The study involved third-year medical students using virtual patient cases as a standard component of their educational program at more than 125 accredited US and Canadian medical schools. The engagement metric algorithm included four equally weighted components of student interactions with the virtual patient. We developed a self-report measure of motivational, emotional, and cognitive engagement and conducted confirmatory factor analysis to assess the validity of the survey responses. We gathered additional validity evidence through educator reviews, factor analysis of the metric, and correlations between student use of the engagement metric and self-report measures of learner engagement. Results: Confirmatory factor analysis substantiated the hypothesized four-factor structure of the survey scales. Educator reviews demonstrated a high level of agreement with content and scoring cut-points (mean Pearson correlation 0.98; mean intra-class correlation 0.98). Confirmatory factor analysis yielded an acceptable fit to a one-factor model of the engagement score components. Correlations of the engagement score with self-report measures were statistically significant and in the predicted directions. Conclusions: We present initial validity evidence for a dynamic online engagement metric based on student interactions in a virtual patient case. We discuss potential uses of such an engagement metric including better understanding of student interactions with online learning, improving engagement through instructional design and interpretation of learning analytics output.

We appreciate the thoughtful responses to our recent article about the role of virtual patients (VPs) in the future of medical education. The letters by Dr. Robison and colleagues and by Dr. White and colleagues suggest that educational technologies may move beyond virtual patients, to virtual humans and virtual families. These are interesting and exciting proposals. Our article addressed several challenges facing medical education, each of which we thought was particularly amenable to VP-based educational strategies. These challenges were in no way meant to be an exhaustive list of challenges facing medical education, or of possible uses of VPs. The letters point to additional challenges, such as patient- and family-centered care, communication skills, and interprofessional education, all of which are very important areas for improvement in medical education. We agree that there is also a place for virtual humans, virtual families, and virtual teams to address these issues. We also suggest that as Dr. Robison and colleagues and Dr. White and colleagues continue their work, they remain focused on developing products that will be used broadly, address important challenges, use sound educational strategies, and result in improved educational outcomes. Ultimately, whether thinking about educational technologies or any other teaching methodology, what really matters is what is learned. With educational technology the possibilities are virtually unlimited, and there is a risk that technological features will result in cognitive overload rather than more learning. As educators try to advance the field by using educational technology, continued attention to educational theory and instructional design will remain critical. Norman B. Berman, MDProfessor, Department of Pediatrics, Geisel School of Medicine at Dartmouth, Hanover, New Hampshire; [email protected] Steven J. Durning, MD, PhDProfessor, Departments of Medicine and Pathology, Uniformed Services University School of Medicine, Bethesda, Maryland. Martin R. Fischer, MD, MMEProfessor and chair for medical education, University Hospital, LMU Munich, Munich, Germany. Soeren Huwendiek, MD, MMESenior lecturer and department head, Department of Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland. Marc M. Triola, MDAssociate professor and associate dean for educational informatics, New York University School of Medicine, New York, New York.

This study explored the use of virtual patient generated data by investigating the association between students’ unprofessional patient summary statements, which they entered during an on-line virtual patient case, and detection of their future unprofessional behavior.

Background: Learning with virtual patients (VPs) is considered useful in medical education for fostering clinical reasoning. As the authoring of VPs is highly demanding, an international exchange of cases might be desirable. However, cultural differences in foreign VPs might hamper learning success.Objective: We investigated the need for support for using VPs from the United States at a German university, with respect to language and cultural differences. Our goal was to better understand potential implementation barriers of a intercultural VP exchange.Methods: Two VPs were presented to 30 German medical students featuring a cultural background different from German standards with respect to diagnostic and therapeutic procedures, ethical aspects, role models, and language (as identified by a cultural adaptation framework). Participants were assigned to two groups: 14 students were advised to complete the cases without further instructions (basic group), and 16 students received written explanatory supplemental information specifically with regard to cultural differences (supplement group). Using a 6-point scale (6=strongly agree), we analyzed the results of an integrated assessment of learning success as well as an evaluation of cases by the students on usefulness for learning and potential issues regarding the language and cultural background.Results: The German students found it motivating to work with cases written in English (6-point scale, 4.5 points). The clinical relevance of the VPs was clearly recognized (6 points), and the foreign language was considered a minor problem in this context (3 points). The results of the integrated learning assessment were similar in both groups (basic 53% [SD 4] vs supplement 52% [ SD 4] correct answers, P=.32). However, students using the supplemental material more readily realized culturally different diagnostic and therapeutic strategies (basic 4 vs supplement 5 points, P=.39) and were less affirmative when asked about the transferability of cases to a German context (basic 5 vs supplement 3 points, P=.048).Conclusions: German students found English VPs to be highly clinically relevant, and they rated language problems much lower than they rated motivation to work on cases in English. This should encourage the intercultural exchange of VPs. The provision of supplemental explanatory material facilitates the recognition of cultural differences and might help prevent unexpected learning effects.

Purpose The ability to create a concise summary statement can be assessed as a marker for clinical reasoning. The authors describe the development and preliminary validation of a rubric to assess such summary statements.Method Between November 2011 and June 2014, four researchers independently coded 50 summary statements randomly selected from a large database of medical students' summary statements in virtual patient cases to each create an assessment rubric. Through an iterative process, they created a consensus assessment rubric and applied it to 60 additional summary statements. Cronbach alpha calculations determined the internal consistency of the rubric components, intraclass correlation coefficient (ICC) calculations determined the interrater agreement, and Spearman rank-order correlations determined the correlations between rubric components. Researchers' comments describing their individual rating approaches were analyzed using content analysis.Results The final rubric included five com ponents: factual accuracy, appropriate narrowing of the differential diagnosis, transformation of information, use of semantic qualifiers, and a global rating. Internal consistency was acceptable (Cronbach alpha 0.771). Interrater reliability for the entire rubric was acceptable (ICC 0.891; 95% confidence interval 0.859-0.917). Spearman calculations revealed a range of correlations across cases. Content analysis of the researchers' comments indicated differences in their application of the assessment rubric.Conclusions This rubric has potential as a tool for feedback and assessment. Opportunities for future study include establishing interrater reliability with other raters and on different cases, designing training for raters to use the tool, and assessing how feedback using this rubric affects students' clinical reasoning skills.

BACKGROUND:Key features examinations (KFEs) have been used to assess clinical decision making in medical education, yet there are no reports of an online KFE-based on a national curriculum for the internal medicine clerkship. What we did: The authors developed and pilot tested an electronic KFE based on the US Clerkship Directors in Internal Medicine core curriculum. Teams, with expert oversight and peer review, developed key features (KFs) and cases.EVALUATION:The exam was pilot tested at eight medical schools with 162 third and fourth year medical students, of whom 96 (59.3%) responded to a survey. While most students reported that the exam was more difficult than a multiple choice question exam, 61 (83.3%) students agreed that it reflected problems seen in clinical practice and 51 (69.9%) students reported that it more accurately assessed the ability to make clinical decisions.CONCLUSIONS:The development of an electronic KFs exam is a time-intensive process. A team approach offers built-in peer review and accountability. Students, although not familiar with this format in the US, recognized it as authentically assessing clinical decision-making for problems commonly seen in the clerkship.

There is great interest in using computer-assisted instruction in medical education, but getting computer-assisted instruction materials used broadly is difficult to achieve. We describe a successful model for the development and maintenance of a specific type of computer-assisted instruction - virtual patients - in medical education. The collaborative model's seven key components are described and compared to other models of diffusion of innovation and curriculum development. The collaborative development model that began in one medical discipline is now extended to two additional disciplines, through partnerships with their respective clerkship director organizations. We believe that the ability to achieve broad use of virtual patients, and to transition the programs from successfully relying on grant funding to financially self-sustaining, resulted directly from the collaborative development and maintenance process. This process can be used in other learning environments and for the development of other types of computer-assisted instruction programs.

PurposeTo explore students' perceptions of virtual patient use in the clinical clerkship and develop a framework to evaluate effects of different integration strategies on students' satisfaction and perceptions of learning effectiveness with this innovation.MethodA prospective, multiinstitutional study was conducted at six schools' pediatric clerkships to assess the impact of integrating Web-based virtual patient cases on students' perceptions of their learning during 2004-2005 and 2005-2006. Integration strategies were designed to meet the needs of each school, and integration was scored for components of virtual patient use and elimination of other teaching methodologies. A student survey was developed, validated, and administered at the end of the clerkship to 611 students. Data were analyzed using confirmatory factor analysis and structural equation modeling.ResultsA total of 545 students (89%) completed the survey. Overall student satisfaction with the virtual patients was high; students reported that they were more effective than traditional methods. The structural model demonstrated that elimination of other teaching methodologies was directly associated with perceived effectiveness of the integration strategies. A higher use score had a significant negative effect on perceived integration, but a positive effect on perceived knowledge and skills gain. Students' positive perceptions of integration directly affected their satisfaction and perception of the effectiveness of their learning.ConclusionsIntegration strategies balancing the use of virtual patients with elimination of some other requirements were significantly associated with students' satisfaction and their perceptions of improved knowledge and skills.

This reflection is based on the premise that clinical education can be improved by more widespread use of computer-assisted instruction (CAI) and that a roadmap will enable more medical educators to begin using CAI. The rationale for CAI use includes many of its inherent features such as incorporation of multimedia and interactivity yet the use of CAI remains limited, apparently because educators are not convinced about the role for CAI. Barriers to CAI use are discussed including misinterpretation of the literature for CAI effectiveness; a disconnect between CAI developers and the educators who make decisions about CAI use; and the paucity of knowledge regarding how to integrate CAI effectively into clinical education. Specific roles for CAI in undergraduate and graduate medical education can include improving uniformity of instruction, providing documentation of exposure or competence, improving the learners’ educational experience or outcomes, and assessment that is matched to learning. Funding for CAI remains an important barrier but the authors believe that this will be overcome when use of CAI becomes more widespread.

Background: The purpose of this research was to determine if students improve interpersonal skills as the third year progresses despite the lack of any specific curriculum or teaching methods.Methods: Third-year students completed I of 3 16-week sequential clerkship blocks. Each student completed a clinical performance examination before and after clerkship consisting of a videotaped standardized patient interview and physical examination. Videotapes were randomly assigned to communication faculty for evaluation.Results: Although the majority (73%) of students improved during their block, 17% showed no improvement, and 12% had deficient interpersonal skills after their clerkship.Conclusions: Despite the lack of skill-directed curriculum, most medical students showed improved interpersonal skill performance after a 16-week clerkship. Developing an interpersonal curriculum for all third-year students may not be necessary. Because faculty are being asked to do more with less, we believe efforts focused on individual students during the third year will be more productive. (c) 2005 Excerpta Medica Inc. All rights reserved.

Methods: The project adhered to four objectives: comprehensive coverage of the core curriculum, uniform approach to CAI pedagogy, multi-institutional development by educators, and extensive evaluation by users. An iterative process of case development separated content from effective use of the software and was followed by peer review. Case and program level evaluation was conducted during project development, and is ongoing with integration of the cases into the curriculum of 6 medical schools.