Background:Residency programs utilize holistic review processes to identify applicants prepared for early clinical responsibility, yet limited evidence links undergraduate medical education (UME) metrics with Accreditation Council for Graduate Medical Education (ACGME) Milestone-based performance. Objective:To determine which UME factors are associated with early pediatric residency performance as measured by the ACGME Milestones. Methods:This multi-institutional, retrospective cohort study included medical students who graduated and completed at least one year of residency in categorical pediatrics between 2014 and 2018. We collected United States Medical Licensing Examination Step 1 and Step 2 Clinical Knowledge scores, overall pediatric clinical performance assessment scores, other required clerkship grades, National Board of Medical Examiners (NBME) Pediatrics Subject Examination scores, leadership roles, research/scholarly activity, and honorary society membership. The primary outcomes were ACGME midyear Milestone data for postgraduate year 1 residents. Linear mixed models with random and fixed effects were used for data analysis. Results:Of 451 eligible graduates from 5 participating sites, 414 (91.8%) were included in the analysis, with Milestone data available for 228 (55.1%). The NBME Examination score was the only covariate associated with all 6 Core Competency Milestones (B=0.03; 95% CI, 0.00-0.06; P<.05). Earning an honors or high pass during the pediatrics clerkship was associated with Milestone performance in 2 competencies-Professionalism and Patient Care (B=1.10-1.18; 95% CI, 0.36-1.92; P<.05). Conclusions:Pediatric NBME scores and clinical performance grades were associated with either all or select midyear Milestone competencies in the first year of pediatrics residency.
Background:Community-based training is crucial for the equitable development of the physician workforce and influences the practice location of new physicians. However, little is known about the distribution of the physicians trained in community-based settings across areas of varying social risks. Objective:To examine the association between community-based training of family physicians and subsequent practice comprising socially disadvantaged populations. Methods:Using the 2017-2022 American Board of Family Medicine Initial Certification Questionnaires and the 2024 American Medical Association Physician Masterfile, we performed a multivariate logistic regression analysis to assess the association between community-based training-applying a broad definition (training outside of large academic centers) and a narrow definition (training specifically in Teaching Health Centers [THCs] or rural/Rural Training Tracks [RTTs])-and subsequent practice location in communities with high health-related social needs. Results:Among the 15 851 family physicians, 5717 (36.1%) trained outside a hospital or large academic center (broad), whereas only 1419 (9%) trained in THCs or rural/RTTs (narrow). Family physicians from a narrowly defined community-based training had higher odds of practicing in socially disadvantaged communities (Social Deprivation Index≥median) compared to their counterparts (OR, 1.224; 95% CI, 1.097-1.366; P<.001). No significant association was found in the model using the broad definition of community-based training. Male, non-White, Hispanic, and international medical graduates were statistically positively associated with practicing in areas with high health-related social needs. Conclusions:This study shows that family physicians who trained in THCs or rural/RTTs are significantly more likely to be practicing in areas with high social needs than those who did not.
Background:Medication abortion using mifepristone and misoprostol is safe and effective, yet few internal medicine (IM) physicians provide this care due to barriers such as stigma, institutional restrictions, and limited training. General deficits in reproductive health training have long been recognized in IM resident training, and gaps remain. Objective:To evaluate the implementation and outcomes of a medication abortion curriculum for IM residents at Cambridge Health Alliance, a Harvard-affiliated program, during the 2022-2023 academic year. Methods:A structured curriculum was developed in partnership between the IM and family medicine departments with 3 components: a 1-2-hour online module, a 70-minute in-person session, and a 3-hour clinical experience at a reproductive health clinic. Data were collected on curriculum participation rates and from online pre- and post-curriculum surveys. Surveys assessed residents' attitudes toward medication abortion, self-reported comfort with counseling and eligibility determination, and knowledge using a brief internally developed assessment. Results:Twenty-three of 24 residents (96%) completed ≥1 components. In the pre-survey, 79% (15 of 19) of residents indicated they would like to provide medication abortion services during residency if trained, and 74% (14 of 19) intended to incorporate it into their future practice. Post-survey results showed significant improvements in comfort with counseling on early abortion options (95% CI, 0.54-1.84; P<.001), determining eligibility (95% CI, 0.61-1.86; P<.001), counseling on medication abortion regimens (95% CI, 0.62-1.89; P<.001), and managing failed abortions (95% CI, 0.50-1.79; P=.001). Conclusions:This curriculum significantly increased IM residents' comfort in providing medication abortion services over 6 months.
Background:The transition from hospital discharge to home is a critical period prone to gaps in care. Telemedicine has potential to smooth this transition; however, few educational interventions assess the unique skills required for high-quality telemedicine care at this critical juncture. Objective:To develop and integrate in situ standardized patient (SP) encounters in internal medicine residents' actual clinics to assess telemedicine skills in the post-discharge period. Methods:We developed 2 cases portraying recently discharged patients, designed behaviorally anchored assessment checklists, created mock electronic health record entries, and scheduled telemedicine visits in residents' clinics throughout the 2023-2024 academic year. SPs assessed skills as "not done," "partly done," or "well done" across 5 skill domains: Information Gathering, Relationship Development, Education and Counseling, Telemedicine-Specific Skills, and Care Transition Skills. We analyzed differences in the percentage of "well done" items, fit an ordinal mixed-effects model to assess for performance by case and postgraduate year (PGY) level, and surveyed residents. Results:All 42 (100%) PGY-1s and PGY-2s in our program participated in 79 total encounters. Residents performed well in core communication domains but struggled with Telemedicine-Specific Skills, Care Transition Skills, and Education and Counseling skills. PGY-2 performance was stronger than PGY-1 performance in these domains. Among residents who completed both cases, performance in case 2, which took place 6 months after case 1, was stronger; this effect was driven by PGY-1 performance. Twenty-nine of 31 residents (94%) reported this intervention improved their telemedicine skills. Conclusions:We report a high-fidelity strategy that captures telemedicine skill development in the context of patient care.
Background:Preference signaling in the residency application process was introduced in the 2021 application cycle and, as of the 2026 residency Match cycle, is used by 27 specialties. Originally intended to help applicants indicate genuine interest in programs, its role has expanded to include mitigating overapplication. Objective:To assess how the number of signals allotted influences application numbers across specialties. Methods:We performed a cross-sectional analysis of US Electronic Residency Application Service (ERAS) data from 22 signaling specialties between 2022 and 2026. Specialties were grouped by number of signals allotted: high (≥20), medium (8-19), low (≤7). The primary outcome was change in average applications per applicant within specialty. The secondary outcome was estimated financial impact using the ERAS tiered fee schedule. Results:Among 70 111 applicants across 22 specialties (high n=6; medium n=8; low n=8), high-signal specialties uniformly reduced applications per applicant (mean, -28.3; range, -47.4 to -2.15). Medium and low tiers showed mixed effects, with an increase in applications per applicant in several specialties. Overall, signaling was associated with an estimated net cost reduction of $7,387,754 across all participating specialties. All high-signal specialties generated cost reduction ($2,991,059 total), whereas 8 medium- and low-signal specialties incurred net cost increases. Conclusions:Substantial reductions in application volume and costs have occurred following implementation of preference signaling, with the high-signal approach reliably leading to a decrease in applications submitted.
Background:Numerous formal and informal resources exist to support the transition to residency. Little is known about how postgraduate year (PGY) 1 residents perceive and utilize these tools. Objective:To explore PGY-1 obstetrics and gynecology (OB/GYN) residents' perspectives on the utility and impact of structured and informal resources supporting the transition to residency. Methods:We used a qualitative thematic analysis of in-depth interviews with PGY-1 OB/GYN residents using self-determination theory as a framework. In 2024, participants were recruited via email and social media from across the United States. Two residents performed virtual interviews. Three independent coders used inductive coding to analyze the data, then interpreted the patterns that emerged using self-determination theory as an interpretive framework. Results:A total of 11 interviews were performed. Three themes were identified in residents' experience that aligned with the core domains of self-determination theory: the heterogeneity of learning needs as new residents strive towards independence (autonomy), responsibilities for patient care serving as a powerful motivator for learning (competence), and the need for a supportive learning environment (belonging). Residents expressed diverse learning needs, and the abundance of tools often contributed to feelings of being overwhelmed. Conclusions:Tailoring support strategies to align with learning goals may enhance the effectiveness of transition resources and promote early resident growth and well-being. Educators may help learners identify resources but also allow for learner autonomy in setting learning goals, focus on practical skills to develop competence in clinical duties, and create structures to support a culture of belonging.
Background:While United States Medical Licensing Examination (USMLE) scores correlate with performance on subsequent standardized tests, it is not known whether they predict success in residency. Nonetheless, they remain the most used metric to narrow the large applicant pool. Objective:To determine whether several filterable metrics reported by the Electronic Residency Application Service (ERAS) predict success in pediatric residency, as measured by Accreditation Council for Graduate Medical Education Pediatrics Milestone (PM) ratings. Methods:Pediatric residency programs participating in the Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network were invited to participate in 2020; 10 of 149 programs (6.7%) provided data on 518 residents. Metrics included type of medical degree, US or non-US medical school, Alpha Omega Alpha and Gold Humanism Honor Society (GHHS) membership, USMLE Step 1 and Step 2 Clinical Knowledge (CK) numerical scores, and Step 2 Clinical Skills first attempt pass/fail status; primary outcome was mean PM ratings across all residency years. Results:GHHS was positively associated with overall PM performance (B=0.15, P=.009), as was Step 2 CK (B=0.06, P=.007). By domain, Step 2 CK was positively associated with Medical Knowledge (MK), Patient Care (PC), and Practice-Based Learning and Improvement (PBLI). GHHS membership was positively associated with MK, PC, PBLI, Professionalism, and Systems-Based Practice. Effect size of GHHS was higher than Step 2 CK across all domains. Conclusions:Among filterable ERAS metrics, GHHS membership showed the strongest association with pediatric residency success, as measured by PM ratings; Step 2 CK had a weaker positive association.
Background:Graduate medical education (GME) leaders face challenges in evaluating program performance due to fragmented data systems and limited resources. Dashboards have been proposed, but few are scalable, participant-informed, and feasible without new infrastructure. Objective:To develop and implement a centralized, participant-informed GME Program Scorecard (GPS) that enables continuous program evaluation and improvement using existing data systems without new data entry. Methods:The GPS was developed at a large academic medical center in 2022-2023. A consensus development panel of 20 GME administrators identified 42 candidate metrics across 13 systems. Program directors (PDs) from the largest residency programs were surveyed (11; 9 responses, 81.8%) to rate each metric's value. The 17 most supported metrics were integrated into a Tableau-based dashboard. The GPS was piloted with 4 programs (3 Accreditation Council for Graduate Medical Education [ACGME]-accredited, 1 non-ACGME-accredited), involving 8 PDs and program coordinators. Feedback was collected via structured forms and a focus group. Metrics were rated on a 1 to 5 scale, with lower scores indicating higher priority. Results:Pre-intervention survey results guided prioritization, with 92.9% of metrics receiving a mean score <3. Pilot users reported high acceptability and identified design refinements incorporated before institutional launch. Development required 2920 analyst hours and $1,600 in licenses. Since launching in January 2024, 309 users (61.4% of 503 potential users) have accessed the GPS with 3933 total views. Conclusions:The GPS demonstrated feasibility, requiring 2920 analyst hours and $1,600 in licenses, and with 61.4% of potential users accessing the dashboard within 22 months of launch.
Background:Summarizing resident assessments for Clinical Competency Committee or mentoring meetings is challenging. This especially applies to raters' free-text comments, which often provide valuable feedback. Objective:To assess feasibility and accuracy of using a large language model (LLM) to label free-text comments as "Strengths" or "Areas for Improvement" from end-of-rotation assessments of residents. Methods:We performed a mixed-methods study of residents' end-of-rotation assessments completed between July 1, 2024 and June 30, 2025. Following data de-identification, we used a private (paid) LLM account (ChatGPT-4o) to compile numerical and label free-text ("Strengths," "Areas for Improvement") feedback. We assessed feasibility using time spent de-identifying data, time for LLMs to return labeled free-text comments, and LLM recall (ability to accurately extract/label free-text feedback). We performed 2 confirmatory trials (free LLMs: MS CoPilot, ChatGPT-4) to ascertain reproducibility. Two subject matter experts labeled free-text comments to determine LLM labeling accuracy. The study was performed in 2025. Results:Our dataset represented 277 individual assessments for 25 unique residents. Feasibility data demonstrated data de-identification time of ∼60 minutes and LLM processing/labeling times of <7 minutes per LLM. All LLMs consistently compiled all free-text entries, accurately labeling 59.7% of free-text comments. Human analysis of comments (476) identified 388 (81.5%) "Strengths" and 88 (18.5%) "Areas for Improvement." Of these, LLM recall was 67.8% for "Strengths" and 23.9% for "Areas for Improvement." Conclusions:We designed prompts that feasibly labeled end-of-rotation assessment free-text comments across 3 LLMs with recall for "Strengths" 2.5 times higher than recall for "Areas for Improvement."
