Abstract In 2009, the Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN), a national educational research network, was formed. We report on evaluation of the network after 10 years of operation by reviewing program context, input, processes, and products to measure its progress in performing educational research that advances training of future pediatricians. Historical changes in medical education shaped the initial development of the network. APPD LEARN now includes 74% (148 of 201) of US Pediatric residency programs and has recently incorporated a network of Pediatric subspecialty fellowship programs. At the time of this evaluation, APPD LEARN had approved 19 member-initiated studies and 14 interorganizational studies, resulting in 23 peer-reviewed publications, numerous presentations, and 7 archived sharable data sets. Most publications focused on how and when interventions work rather than whether they work, had high scores for reporting rigor, and included organizational and objective performance outcomes. Member program representatives had positive perceptions of APPD LEARN’s success, with most highly valuing participation in research that impacts training, access to expertise, and the ability to make authorship contributions for presentations and publication. Areas for development and improvement identified in the evaluation include adopting a formal research prioritization process, infrastructure changes to support educational research that includes patient data, and expanding educational outreach within and outside the network. APPD LEARN and similar networks contribute to high-rigor research in pediatric education that can lead to improvements in training and thereby the health care of children.

OBJECTIVES The demands of residency training may impact trainees’ decision to have children. We examined characteristics of pediatric residents’ decisions regarding childbearing, determinants of resident parental leave, and associations with well-being. METHODS A survey of 845 pediatric residents at 13 programs was conducted between October 2019 and May 2020. Survey items included demographics, desire for future children, and logistics of parental leave. Outcomes included parental leave length, burnout and depression screening results, satisfaction with duration of breastfeeding, and satisfaction with parental leave and parenthood decisions. RESULTS Seventy-six percent (639 of 845) of residents responded to the survey. Fifty-two percent (330) of respondents reported delaying having children during residency, and 29% (97) of those were dissatisfied with their decision to do so. Busy work schedule (89.7%), finances (50.9%), and a desire not to extend residency (41.2%) were the most common reasons for delay. Of respondents, 16% were parents and 4% were pregnant or had pregnant partners. Sixty-one parental leaves were reported, and 67% of parents reported dissatisfaction with leave length. The most frequently self-reported determinant of leave duration was the desire not to extend residency training (74%). Program mean leave length was negatively associated with burnout, measured as a dichotomous outcome (odds ratio = 0.81 [95% confidence interval 0.68–0.98]; P = .02). CONCLUSIONS Many pediatric trainees delay parenthood during residency and are not satisfied with their decision to do so. Pediatric resident parental leave remains short and variable in duration, despite the positive association between longer leaves and overall well-being.

Purpose To determine which narrative performance level for each general pediatrics entrustable professional activity (EPA) reflects the minimum level clinical competency committees (CCCs) felt should be associated with graduation as well as initial entrustment and compare expected narrative performance levels (ENPLs) for each EPA with actual narrative performance levels (ANPLs) assigned to residents at initial entrustment. Method A series of 5 narratives, corresponding to the 5 milestone performance levels, were developed for each of the 17 general pediatrics EPAs. In academic year (AY) 2015-2016, the CCCs at 22 Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network member sites reported ENPLs for initial entrustment and at time of graduation. From AYs 2015-2016 to 2017-2018, programs reported ANPLs for initial entrustment decisions. ENPLs and ANPLs were compared using a logistic mixed effects model. Results ENPLs for graduation and entrustment were most often level 3 (competent) followed by level 4 (proficient). For 8 EPAs, the ENPLs for graduation and entrustment were the same. For the remaining 9, some programs would entrust residents before graduation or graduate them before entrusting them. There were 4,266 supervision level reports for initial entrustment for which an ANPL was provided. ANPLs that were lower than the ENPLs were significantly more likely to be assigned to the medical home-well child (OR = 0.39; 95% CI: 0.26-0.57), transition to adult care (OR = 0.43; 95% CI: 0.19-0.95), behavioral or mental health (OR = 0.36; 95% CI: 0.18-0.71), make referrals (OR = 0.31; 95% CI: 0.17-0.55), lead a team (OR = 0.34; 95% CI: 0.22-0.52), and handovers (OR = 0.18; 95% CI: 0.09-0.36) EPAs. Conclusions CCCs reported lower ENPLs for graduation than for entrustment for 5 EPAs, possibly indicating curricular gaps that milestones and EPAs could help identify.

Educational research networks leverage shared goals and common infrastructure to overcome traditional barriers to medical education research, including small sample sizes, lack of generalizability, need for expertise in statistical analysis, and limitations on data sharing. The diversity of extant network models today is exciting and provides a set of common options and challenges that newly emerging networks can expect. These include decisions about network focus, organization of data, sampling strategies, funding, and governance. Common challenges include managing authorship, human subjects protection rules, data use agreements, and statistical disclosure control. Medical education research networks both advance the field and develop the researchers who participate in them. The authors repeat the call that they and others have made for the development of networks to promulgate best practices and coordinate multinetwork (multinational, multispecialty, and cross-curriculum) studies.

Question What is the progression of performance for entrustable professional activities (EPAs) throughout pediatric residency training and at graduation? Findings This multisite cohort study of 1987 pediatric residents found that developmental growth curves can be established for EPAs. When generated to reflect the results in this study, at least 90% of trainees achieved the level of unsupervised practice at the end of residency for only 8 of the 17 EPAs studied. Meaning This study suggests that gaps exist between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve based on the general pediatrics EPAs. Importance Entrustable professional activities (EPAs) are an emerging workplace-based, patient-oriented assessment approach with limited empirical evidence. Objective To measure the development of pediatric trainees' clinical skills over time using EPA-based assessment data. Design, Setting, and Participants Prospective cohort study of categorical pediatric residents over 3 academic years (2015-2016, 2016-2017, and 2017-2018) assessed on 17 American Board of Pediatrics EPAs. Residents in training at 23 pediatric residency programs in the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network were included. Assessment was conducted by clinical competency committee members, who made summative assessment decisions regarding levels of supervision required for each resident and each EPA. Data were collected from May 2016 to November 2018 and analyzed from November to December 2018. Interventions Longitudinal, prospective assessment using EPAs. Main Outcomes and Measures Trajectories of supervision levels by EPA during residency training and how often graduating residents were deemed ready for unsupervised practice in each EPA. Results Across the 5 data collection cycles, 1987 residents from all 3 postgraduate years in 23 residency programs were assigned 25503 supervision level reports for the 17 general pediatrics EPAs. The 4 EPAs that required the most supervision across training were EPA 14 (quality improvement) on the 5-level scale (estimated mean level at graduation, 3.7; 95% CI, 3.6-3.7) and EPAs 8 (transition to adult care; mean, 7.0; 95% CI, 7.0-7.1), 9 (behavioral and mental health; mean, 6.6; 95% CI, 6.5-6.6), and 10 (resuscitate and stabilize; mean, 6.9; 95% CI, 6.8-7.0) on the expanded 5-level scale. At the time of graduation (36 months), the percentage of trainees who were rated at a supervision level corresponding to "unsupervised practice" varied by EPA from 53% to 98%. If performance standards were set to align with 90% of trainees achieving the level of unsupervised practice, this standard would be met for only 8 of the 17 EPAs (although 89% met this standard for EPA 17, performing the common procedures of the general pediatrician). Conclusions and Relevance This study presents initial evidence for empirically derived practice readiness and sets the stage for identifying curricular gaps that contribute to discrepancy between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve. Future work should compare these findings with postgraduation outcomes data as a means of seeking validity evidence. This cohort study measures the development of pediatric resident clinical skills using assessments based on entrustable professional activities.

OBJECTIVE: This single-specialty, multi-institutional study aimed to determine 1) the association between milestone ratings for individual competencies and average milestone ratings (AMRs) and 2) the association between AMRs and recommended supervisory role categorizations made by individual clinical competency committee (CCC) members. METHODS: During the 2015-16 academic year, CCC members at 14 pediatric residencies reported milestone ratings for 21 competencies and recommended supervisory role categories (may not supervise, may supervise in some settings, may supervise in all settings) for residents they reviewed. An exploratory factor analysis of competencies was conducted. The associations among individual competencies, the AMR, and supervisory role categorizations were determined by computing bivariate correlations. The relationship between AMRs and recommended supervisory role categorizations was examined using an ordinal mixed logistic regression model. RESULTS: Of the 155 CCC members, 68 completed both milestone assignments and supervision categorizations for 451 resi- dents. Factor analysis of individual competencies controlling for clustering of residents in raters and sites resulted in a single -factor solution (cumulative variance: 0.75). All individual competencies had large positive correlations with the AMR (correlation coefficient: 0.84-0.93), except for two professionalism competencies (Profl : 0.63 and Prof4: 0.65). When combined across training year and time points, the AMR and supervisory role categorization had a moderately positive correlation (0.56). CONCLUSIONS: This exploratory study identified a modest correlation between average milestone ratings and supervisory role categorization. Convergence of competencies on a single factor deserves further exploration, with possible rater effects warranting attention.

Purpose Entrustment has become a popular assessment framework in recent years. Most research in this area has focused on how frontline assessors determine when a learner can be entrusted. However, less work has focused on how these entrustment decisions are made. The authors sought to understand the key factors that pediatric residency program clinical competency committee (CCC) members consider when recommending residents to a supervisory role. Method CCC members at 14 pediatric residency programs recommended residents to one of five progressive supervisory roles (from not serving as a supervisory resident to serving as a supervisory resident in all settings). They then responded to a free-text prompt, describing the key factors that led them to that decision. The authors analyzed these responses, by role recommendation, using a thematic analysis. Results Of the 155 CCC members at the participating programs, 84 completed 769 supervisory role recommendations during the 2015–2016 academic year. Four themes emerged from the thematic analysis: (1) Determining supervisory ability follows from demonstrated trustworthiness; (2) demonstrated performance matters, but so does experience; (3) ability to lead a team is considered; and (4) contextual considerations external to the resident are at play. Conclusions CCC members considered resident and environmental factors in their summative entrustment decision making. The interplay between these factors should be considered as CCC processes are optimized and studied further.

OBJECTIVE: To explore justifications for differences between summative entrustment decisions made about pediatric residents by individuals who are charged with the review of residents (clinical competency committee, or CCC, members) and those who ultimately make final summative decisions about resident performance (program directors, or PDs). METHODS: Individual CCC member and PD supervisory role categorizations were made in the 2015 to 2016 academic year at 14 pediatric residency programs, placing residents into 1 of 5 progressive supervisory roles. When PD recommendations differed from CCC members, a free-text justification was requested. Free-text responses were analyzed using manifest content analysis. RESULTS: In total, 801 supervisory role categorizations were made by both CCC members and PDs, with the same recommendations made in 685 cases. In the 116 instances of discrepancy, PDs assigned a lower level of supervisory responsibility (n = 73) more often than a greater one (n = 43). When moving residents to a greater supervisory role category, PDs had more justifications anchored in resident performance than experience. When moving residents to a lower supervisory role categorization, PDs conversely noted experience more than performance. CONCLUSIONS: PDs provide more justifications anchored in resident performance when moving residents to a greater supervisory role category compared with CCC members. However, when moving residents to a lower supervisory role categorization, they note experience more than performance. These patterns may or may not be entirely consistent with a competency-based approach and should be explored further.

BACKGROUND:Clinical Competency Committees (CCCs) are charged with making summative assessment decisions about residents.OBJECTIVE:We explored how review processes CCC members utilize influence their decisions regarding residents' milestone levels and supervisory roles.METHODS:We conducted a multisite longitudinal prospective observational cohort study at 14 pediatrics residency programs during academic year 2015-2016. Individual CCC members biannually reported characteristics of their review process and Accreditation Council for Graduate Medical Education milestone levels and recommended supervisory role categorizations assigned to residents. Relationships among characteristics of CCC member reviews, mean milestone levels, and supervisory role categorizations were analyzed using mixed-effects linear regression, reported as mean differences with 95% confidence intervals (CIs), and Bayesian mixed-effects ordinal regression, reported as odds ratios (ORs) and 95% credible intervals (CrIs).RESULTS:A total of 155 CCC members participated. Members who provided milestones or other professional development feedback after CCC meetings assigned significantly lower mean milestone levels (mean 1.4 points; CI -2.2 to -0.6; P < .001) and were significantly less likely to recommend supervisory responsibility in any setting (OR = 0.23, CrI 0.05-0.83) compared with CCC members who did not. Members recommended less supervisory responsibility when they reviewed more residents (OR = 0.96, 95% CrI 0.94-0.99) and participated in more review cycles (OR = 0.22, 95% CrI 0.07-0.63).CONCLUSIONS:This study explored the association between characteristics of individual CCC member reviews and their summative assessment decisions about residents. Further study is needed to gain deeper understanding of factors influencing CCC members' summative assessment decisions.

Background: Clinical competency committee (CCC) identification of residents with performance concerns is critical for early intervention.Methods: Program directors and 94 CCC members at 14 pediatric residency programs responded to a written survey prompt asking them to describe how they identify residents with performance concerns. Data was analyzed using thematic analysis.Results: Six themes emerged from analysis and were grouped into two domains. The first domain included four themes, each describing a path through which residents could meet or exceed a concern threshold:1) written comments from rotation assessments are foundational in identifying residents with performance concerns, 2) concerning performance extremes stand out, 3) isolated data points may accumulate to raise concern, and 4) developmental trajectory matters. The second domain focused on how CCC members and program directors interpret data to make decisions about residents with concerns and contained 2 themes: 1) using norm- and/or criterion-referenced interpretation, and 2) assessing the quality of the data that is reviewed.Conclusions: Identifying residents with performance concerns is important for their education and the care they provide. This study delineates strategies used by CCC members across several programs for identifying these residents, which may be helpful for other CCCs to consider in their efforts.