ObjectivesCommunication and other clinical skills are routinely assessed in medical schools using Objective Structured Clinical Examinations (OSCEs) so routinely that it can be difficult to monitor and maintain validity. We report on the accumulation of validity evidence for the Clinical Communication Skills Assessment Tool (CCSAT) based on its use with 9 cohorts of medical students in a high stakes OSCE.MethodsWe describe the implementation of the CCSAT including information on the underlying model, the tool’s items, domains, scales and scoring, and its role in curriculum. Internal structure is explored through item, internal consistency, and confirmatory factor analyses. Evidence for CCSAT validity is synthesized within prevailing frameworks (Messick12 and Kane13) based on continuous quality improvement and use of the CCSAT for feedback, remediation, curricular design, and research.ResultsImplementation of the CCSAT over time has facilitated our communication skills curriculum and training. Thoughtful case development and investment in standardized patient training has contributed to data quality. Item analysis supports our behaviorally anchored scale (not done, partly and well done) and the skills domains suggested by an a priori evidence-based clinical communication model were confirmed via analysis of actual student data. Evidence synthesized across the frameworks suggests consistent validity of the CCSAT for generalization inferences (that it captures the construct), responsiveness (sensitivity to change/difference), content validity/internal structure, relationships to other variables, and consequences/implications. More evidence is needed to strengthen validity of CCSAT scores for understanding extrapolation inferences and real-world implications.Conclusions and Practice ImplicationsThis pragmatic approach to evaluating validity within a program of assessment serves as a model for medical schools seeking to continuously monitor the quality of clinical skill assessments, a need made particularly relevant since the US NBME no longer requires the Step 2 Clinical Skills exam, leaving individual schools with the responsibility for ensuring graduates have acquired the requisite core clinical skills. We document strong evidence for CCSAT validity over time and across cohorts as well as areas for improvement and further examination.

PurposeAs the COVID-19 pandemic forced most colleges and universities to go online, student health centers rapidly shifted to telehealth platforms without frameworks for virtual care provision. An urban student health center implemented a needs assessment involving unannounced standardized patients (USPs) to evaluate the integration of a new telehealth workflow and clinicians' virtual communication skills.MethodsFrom April to May 2021, USPs conducted two video visits with 12 primary care and four women's health clinicians (N = 16 clinicians; 32 visits). Cases included (1) a 21-year-old female presenting for birth control with a positive Patient Health Questionaire-9 and (2) a 21-year-old male, who vapes regularly, with questions regarding safe sex with men. Clinicians were evaluated using a checklist completed by the USP immediately following the visit and a systematic chart review of the electronic health record.ResultsUSP feedback indicates most clinicians received high ratings for general communication skills but may benefit from educational intervention in several key telemedicine skills. Clinicians struggled with using nonverbal signals to enrich communication (47% well done), acknowledging emotions (34% well done), and using video for information gathering (34% well done). Low rates of standard screenings (e.g., 63% administered the PHQ-2, <50% asked about alcohol use) suggested protocols for in-person care were not easily incorporated into telehealth practices, and clinicians may benefit from enhanced care team support. Performance reports were shared with clinicians and leadership postvisit.DiscussionResults suggest project design and implementation is scalable and feasible for use at other institutions, offering a structured methodology that can improve general student health care.

Background: The availability of faculty proficient in point of care ultrasound (POCUS) has emerged as a barrier to the ongoing expansion of POCUS across the field of internal medicine. We sought to examine the faculty graduates of our institutional POCUS program to identify characteristics associated with long-term proficiency to inform curricula and guide institutional support. Methods: We emailed a test and survey to the 89 faculty graduates of the annual POCUS course we have run from 2018-2023. Results: Of the 46 participants (52%) who completed the test and survey, the overall median test score was 72%. Graduates were most confident with image acquisition of the lung, and were most likely to use ultrasound to evaluate ascites and dyspnea. All 11 participants reporting completion of an image portfolio were actively teaching POCUS, whereas only 54% of non-completers were teaching. Portfolio completers scored higher on the test compared to non-completers (median 92% and 68%, respectively, p <0.01) and were more confident in image acquisition and interpretation (p<0.001). Conclusions: In this long-term, single-institutional study of faculty graduates of an annual POCUS program, those who reported completing an image portfolio scored significantly higher on a knowledge test, reported higher confidence with image acquisition and interpretation, and reported using and teaching POCUS more frequently compared to graduates who did not complete the portfolio. POCUS education programs should be designed to foster continuous scanning practice and image portfolio completion.