In this commentary, the authors respond to Ellaway & Patocka's "To define or not to define: A commentary on the Case for Metacognitive Reflection," which raised several thoughtful and discerning questions concerning the implications of defining, or alternatively, refraining from defining constructs. The authors advocate for navigating the liminal space between precision and flexibility, acknowledging the dynamic and permeable nature of conceptual boundaries while striving for clarity. Moreover, they emphasize that only after a boundary is described can the tensions, gaps, exceptions, and contradictions around that boundary be explored. Finally, the authors highlight the significance of contextual definitions, fostering shared understanding, and embracing abductive reasoning (when is becomes as) to promote dialogue and advance knowledge in health professions education.

IntroductionTest-enhanced learning (TEL) is an impactful teaching and learning strategy that prioritises active learner engagement through the process of regular testing and reviewing. While it is clear that meaningful feedback optimises the effects of TEL, the ideal timing of this feedback (i.e. immediate or delayed) in a medical education setting is unclear.MethodForty-one second-year medical students were recruited from the University of Melbourne. Participants were given a multiple-choice question test with a mix of immediate (i.e. post-item) and delayed (i.e. post-item-block) conceptual feedback. Students were then tested on near and far transfer items during an immediate post-test, and at a one-week follow-up.ResultsA logistic mixed effects model was used to predict the probability of successful near and far transfer. As expected, participants in our study tended to score lower on far transfer items than they did on near transfer items. In addition, correct initial response on a parent question predicted subsequent correct responding. Contrary to our hypotheses, the feedback timing effect was non-significant-there was no discernible difference between feedback delivered immediately versus delayed feedback.DiscussionThe findings of this study suggest that the timing of feedback delivery (post-item versus post-item-block) does not influence the efficacy of TEL in this medical education setting. We therefore suggest that educators may consider practical factors when determining appropriate TEL feedback timing in their setting.

Purpose: To support and explore medical students’ professional identity formation1 as a physician through experiential learning in authentic clinical environments, which occurs during their preclinical years, allowing for earlier association of preclinical knowledge to patient-specific contexts and earlier engagement with threshold concepts2 essential to becoming a physician. Approach: Innovation: Throughout phase 1 of Washington University School of Medicine’s (WUSM’s) new Gateway Curriculum, preclinical students complete three, 3-week clinical immersions. There are 3 types of immersions—ambulatory/emergency department, procedural/surgical, and inpatient, across a variety of specialties. Incorporating experiential learning theory,3 students perform clinical activities like those of clerkships—taking medical histories, performing physical exams, presenting patients, calling consults, writing notes, and participating in the operating room, as well as activities focused on course objectives such as incorporation within an interprofessional team, talking with patients about their lived experiences, and discussing the impact of health systems on patients. Furthermore, activity cards have students record clinical practice activities related to course objectives. Assessment of students within each immersion occurs via direct observation and longitudinal clinical performance assessments, evidence-based medicine critical appraisals, and analytic writing assignments. These writing assignments require students to demonstrate understanding of rotation objectives based on their lived clinical experiences. Writing prompts for these assignments ask students to discuss course content related to objectives, such as interprofessional care teams, systems-based care, social determinants of health, and personal learning goals developed through reflective use of feedback. Evaluation: Students evaluate immersions through satisfaction ratings and open-text comments on end-of-immersion questionnaires. WUSM’s Program Evaluation/Continuous Quality Improvement Unit (PECQI) summarized ratings and categorized open-text comments on the questionnaires. A separate team of 3 evaluators, who were not part of PECQI and not involved in planning or implementing the immersions, analyzed a sample of the analytic writing assignments from all 3 types of immersions using conventional content analysis.4 Outcomes: The first cohort in the Gateway Curriculum includes 105 students, all of whom completed all 3 immersions and associated assignments. Qualitative analysis of a sample of data, obtained at completion of the students’ third immersion, shows that students consistently describe their role as a liminal state from a passive observer—self-identified as a student trait—to an increasingly active care provider—a trait associated with the physician’s role. Students comment on discomfort and growth of this transitional stage, noticing, for example, the benefits and flaws of health care systems as they interact with social determinants of health, the interprofessional care team, patients, and their patients’ families. Students identify the value of communication, partnering with patients, and learning from interprofessional team members. In aspirational statements, students begin to self-identify threshold concepts central to their future identity as a physician such as patient-centeredness, the importance of equitable care, comfort with ambiguity, and need for lifelong learning. Quantitative data show that students feel that critical elements of immersions occur and that students self-identified “working in interprofessional teams” and “opportunities to practice clinical care” as well-executed aspects of the immersions. The analytic writing assignment prompts provide a lens for professional development, encouraging the student to reflect on their transition from student to physician in multiple contexts. Qualitative data from the written assignments show that these intentionally designed clinical immersions coupled with longitudinal, guided self-reflection expose medical students to key threshold concepts of physician professional identity formation. Significance: As a direct result of their participation within clinical interprofessional teams, combined with the directed learning of clinical immersions, preclinical students begin to frame their remaining preclinical activities through the lens of their identity as a future physician. Therefore, students enter their clerkship years having already begun their development of a professional identity as a physician.

Introduction The role of feedback in test-enhanced learning is an understudied area that has the potential to improve student learning. This study investigates the influence of different forms of post-test feedback on retention and transfer of biomedical knowledge within a test-enhanced learning framework. Methods 64 participants from a Canadian and an Australian medical school sat two single-best-answer formative multiple choice tests one week apart. We compared the effects of conceptually focused, response-oriented, and simple right/wrong feedback on a learner’s ability to correctly answer new (transfer) questions. On the first test occasion, participants received parent items with feedback, and then attempted items closely related (near transfer) to and more distant (far transfer) from parent items. In a repeat test at 1 week, participants were given different near and far transfer versions of parent items. Feedback type, and near and far transfer items were randomized within and across participants. Results Analysis demonstrated that response-oriented and conceptually focused feedback were superior to traditional right/wrong feedback for both types of transfer tasks and in both immediate and final retention test performance. However, there was no statistically significant difference between response-orientated and conceptually focused groups on near or far transfer problems, nor any differences in performance between our initial test occasion and the retention test 1 week later. As with most studies of transfer, participants’ far transfer scores were lower than for near transfer. Discussion Right/wrong feedback appears to have limited potential to augment test-enhanced learning. Our work suggests that item-level feedback and feedback that identifies and elaborates on key conceptual knowledge are two important areas for future research on learning, retention and transfer.

May 6, 2019April 9, 2019Free AccessJudgements of Competence: Negotiated Meaning Making in Resident Evaluations (P2.9-004)Douglas Larsen, Roxann Lawrence, and Lara VarpioAuthors Info & AffiliationsApril 9, 2019 issue92 (15_supplement)https://doi.org/10.1212/WNL.92.15_supplement.P2.9-004 Letters to the Editor

Larsen encourages educators and researchers to expand their definition of learning beyond cognitive processes to include practices that emerge from social interactions and system changes.

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited. Cultural historical activity theory (CHAT) provides a framework to analyze complex systems both in research and in curriculum design and evaluation. By using CHAT, scholars and leaders can study new educational innovations or existing processes to discover the facilitators of and barriers to change.1 CHAT emerges from Vygotsky’s concept of object-oriented action (an activity) mediated by tools (both symbolic [e.g., competencies] and physical [e.g., a patient’s chart]).2 Engeström, a prominent CHAT theorist, uses the activity system as the unit of analysis to explore the myriad relationships that are part of a complex setting.3 CHAT can be used to understand systems of different scales—from daily interpersonal interactions to large institutional initiatives.

Educational systems are rarely designed for long-term retention of information. Strong evidence has emerged from cognitive psychology and applied education studies that repeated retrieval of information significantly improves retention compared to repeated studying. This effect likely emerges from the processes of memory consolidation and reconsolidation. Consolidation and reconsolidation are the means by which memories are organized into associational networks or schemas that are created and recreated as memories are formed and recalled. As educators implement retrieval practice, they should consider how various test formats lead to different degrees of schema activation. Repeated acts of retrieval provide opportunities for schemas to be updated and strengthened. Spacing of retrieval allows more consolidated schemas to be reactivated. Feedback provides metacognitive monitoring to ensure retrieval accuracy and can lead to shifts from ineffective to effective retrieval strategies. By using the principles of retrieval practice, educators can improve the likelihood that learners will retain information for longer periods of time.

Exome sequencing has markedly enhanced the discovery of genes implicated in Mendelian disorders, particularly for individuals in whom a known clinical entity could not be assigned. This has led to the recognition that phenotypic heterogeneity resulting from allelic mutations occurs more commonly than previously appreciated. Here, we report that missense variants in CDC42, a gene encoding a small GTPase functioning as an intracellular signaling node, underlie a clinically heterogeneous group of phenotypes characterized by variable growth dysregulation, facial dysmorphism, and neurodevelopmental, immunological, and hematological anomalies, including a phenotype resembling Noonan syndrome, a developmental disorder caused by dysregulated RAS signaling. In silico, in vitro, and in vivo analyses demonstrate that mutations variably perturb CDC42 function by altering the switch between the active and inactive states of the GTPase and/or affecting CDC42 interaction with effectors, and differentially disturb cellular and developmental processes. These findings reveal the remarkably variable impact that dominantly acting CDC42 mutations have on cell function and development, creating challenges in syndrome definition, and exemplify the importance of functional profiling for syndrome recognition and delineation.

This supplement includes 11 research papers accepted by the 2018 Research in Medical Education Committee. In this Commentary, the authors draw the attention of medical educators and researchers to questions about the researcher's philosophical paradigm. Such questions fundamentally guide research-the choice of conceptual frameworks, methods, and methodology. The authors provide a brief overview of research paradigms and the related concept of axiology and praxeology in medical education research. The authors map the 11 accepted research articles, describing the researchers' stated or implied worldviews and the impact on chosen methods. The authors close by encouraging researchers to state the research paradigm behind their research, comment on how they ensured that the methods used displayed consistency with that paradigm, and highlight the value the research adds to everyday education.

Many consider well-developed self-regulated learning skills a critical outcome in medical education.1 The Accreditation Council for Graduate Medical Education core competency of practice-based learning and improvement and its associated milestones codify graduate medical education's commitment to this principle.2,3 Residents must demonstrate their ability to create and act on learning and improvement goals. Often this occurs through individualized learning plans.2,4 However, these efforts do not always achieve their intended outcomes. For instance, in a national survey of pediatrics and medicine-pediatrics residents, only 39% felt that setting goals through individualized learning plans was worth the time and effort, and only 26% of those residents tracked their progress with their learning goals.5 In another study, 35% of residents reported that individualized learning plans provided a useful framework for learning, and 30% felt that their goals and plans improved their awareness of the learning process.6 Other studies have shown that residents value opportunities to direct their own learning, that they struggle to manage their learning, and that they desire guidance on directing their learning.7Clearly, with self-regulated learning, we need to understand how to support and educate our trainees more effectively. In this issue of the Journal of Graduate Medical Education, Saddawi-Konefka and colleagues8 provide medical educators with an important piece of that puzzle. In this study, investigators use the WOOP (Wish, Outcome, Obstacle, Plan) training technique to increase residents' accomplishment of their personal study goals. As the authors note, WOOP is a memorable acronym used to communicate the principles of mental contrasting with implementation intentions (MCII) from the psychology literature. The authors demonstrated that by training residents to identify potential obstacles in achieving their study goals, and creating a plan to overcome those obstacles, they almost tripled the amount of goal-directed study time compared with controls. Significantly, the investigators used a robust control group in which residents were instructed on the characteristics of good goals, as would typically happen in most training programs. In contrast to many educational studies, the authors demonstrate a change in behavior, not simply increased acceptance of an intervention or greater confidence in ability. An important aspect of this study is that WOOP (as a self-regulated learning technique) is not time or resource intensive, and it can easily be implemented into residency training programs.In a commonly used cyclical model of self-regulated learning, individuals (1) make goals and plans; (2) monitor their actions to ensure that they are achieving their goals; and (3) reflect on the outcomes of their actions to determine which new goals they will create.9 Use of MCII (or WOOP) helps to influence the transition from goal setting to self-monitored action and improves the efficacy of that self-monitoring. Mental contrasting consists of identifying a desired future state, reflecting on the greatest benefits of obtaining the goal, and contrasting that to barriers in the current reality that may prevent that state from being obtained.10 When a person feels that achieving the goal is highly likely, this contrasting exercise has been found to energize the person and greatly increase her or his commitment to the goal.11 This process establishes a strong link in the mind of learners between the obstacles to achieving their goals and the behaviors necessary to overcome the obstacles.12 Contemplating the desired future state without contrasting it to the challenges of reality actually decreases a person's energy (ie, the level of activation or invigoration leading to goal commitment).13 Implementation intentions involve creating specific “if-then” plans to overcome anticipated obstacles.14 In this way, the person monitors his or her activity and takes corrective action once he or she encounters given situations. Mental contrasting combined with implementation intentions has been shown to be more effective than using either alone.15 Use of MCII has been shown to change a variety of behaviors—from decreasing unhealthy snacking to improving school activity management in children with attention deficit hyperactivity disorder.15,16The MCII/WOOP approach succeeds by making a strong connection between the goal and the plan to achieve it. Typically, residents focus more on the goal itself rather than how to achieve it. Results of previous studies highlight that creating and then implementing a plan are the most difficult aspects of goal setting and striving for residents.17 The distinction between focusing on the characteristics of the goal versus the plan to achieve the goal is well illustrated in the current study. While the control group was trained to create specific and actionable goals, the WOOP intervention group was trained to focus on the plan to achieve their goals. In another study of self-regulated learning, strategic planning—more than goal setting—was found to be the element that most strongly correlated with academic success.18The role of time frames is often neglected in the discussion of self-regulated learning in residency education. In the present study, residents set goals that applied to their immediate rotation. In our experience, residency programs often require trainees to create individualized learning plans and goals that span longer periods of time. Goals and plans created in this time frame may be difficult to incorporate into daily practice and may be easily forgotten. Other research has shown that goals created around a short time frame improved learners' awareness of their thoughts and actions.19 Further research in medical education settings is needed to clarify whether WOOP is effective with goals that span long periods of time, or whether it produces its greatest effects with goals that are achieved over short time frames.The technique modeled by Saddawi-Konefka and colleagues is likely to be only a piece of the larger picture as educators develop tools to facilitate self-regulation. Other models of self-regulated learning have emphasized the importance of the social context.1 Residents identify institutional support and faculty engagement as the 2 most important factors that determine whether their formal written goals are effective.20 This support comes as programs provide the tools for creating and following up on goals, and as faculty actively participate with the residents in their goal setting and attainment. As noted by the authors, the goals in the current study are relatively simple and rely solely on the actions of the residents. It remains to be seen how MCII/WOOP may affect more complex process-oriented and patient care–focused goals. These types of goals may require a greater consideration of the social and collaborative nature of practice. When analyzed from a sociomaterial perspective, written learning goals have been shown to serve as tools to facilitate collaboration between learners and their supervisors.21 Learning goals used in this manner cause both learners and supervisors to act in new ways and, thus, expand their practice experiences.Self-regulated learning is a complex activity that requires a sophisticated and multilayered approach in our educational work. For educators to move beyond regulatory requirements and checkbox assessments to actually change the behavior and practices of learners, we must have tools that address self-regulated learning from the level of the individual's cognitive processes, as well as the social and material interactions that also determine learners' actions. The emerging research in this area shows exciting opportunities, as educators seek to put together the complex puzzle of optimal self-regulated learning and practice.

CONTEXT Learning goal programmes are often created to help students develop self-regulated learning skills; however, these programmes do not necessarily consider the social contexts surrounding learning goals or how they fit into daily educational practice.OBJECTIVES We investigated a high-frequency learning goal programme in which students generated and shared weekly learning goals with their clinical teams in core Year 3 clerkships. Our study explores: (i) how learning goals were incorporated into the clinical work, and (ii) the factors that influenced the use of students' learning goals in work-based learning.METHODS We conducted semi-structured interviews with 14 students and 14 supervisors (attending physicians and residents) sampled from all participating core clerkships. Interviews were coded for emerging themes. Using cultural historical activity theory and knotworking as theoretical lenses, we developed a model of the factors that influenced students' learning goal usage in a work-based learning context.RESULTS Students and supervisors often faced the challenge of reconciling contradictions that arose when the desired outcomes of student skill development, grading and patient care were not aligned. Learning goals could function as tools for developing new ways of acting that overcame those contradictions by facilitating collaborative effort between students and their supervisors. However, for new collaborations to take place, both students and supervisors had to engage with the goals, and the necessary patients needed to be present. When any one part of the system did not converge around the learning goals, the impact of the learning goals programme was limited.CONCLUSIONS Learning goals are potentially powerful tools to mediate interactions between students, supervisors and patients, and to reconcile contradictions in work-based learning environments. Learning goals provide a means to develop not only learners, but also learning systems.

PROBLEM:Although self-regulated learning (SRL) is considered a fundamental skill that must be developed in physician training, many programs of SRL utilize learning goals that are generated only at the beginning of learning experiences or are widely spaced apart in time. These goals are often not formally shared with those actually working with the learner in the clinical setting.INTERVENTION:We developed a program of written, student-generated weekly learning goals in which students focused on processes of becoming better doctors for their patients. These goals were shared at the beginning of each week with students' clinical teams for feedback and incorporation into the work.CONTEXT:The weekly learning goals program was developed and implemented as part of a required 3rd-year neurology clerkship. At the end of each 4-week clerkship, students were asked to evaluate the program through an anonymous electronic survey utilizing quantitative and open-ended qualitative questions.OUTCOME:Seventy-six of 82 students completed the evaluation survey (93% response rate). Eighty-six percent reported that the weekly learning goals increased their awareness of their thoughts and actions. Seventy-eight percent reported that the learning goals helped to improve their clinical performance to some degree, and 57% reported that the learning goals increased their focus on patient care. Students described a greater sense of focus on self-assessment and accountability from their goals. Students often commented that engagement from attendings and residents regarding their goals was a key element for successful learning from their goals.LESSONS LEARNED:Student-generated weekly learning goals on a neurology clerkship appear to be an effective method to operationalize SRL. For most students, the frequency of the goals allowed for close self-monitoring, and the act of sharing goals with the team opened a new avenue for dialogue between students and their supervisors.

Objective: Using qualitative research methods, we identified factors that influence the efficacy of students' weekly learning goals in clinical education. Background: Learning goals are an essential part of self-regulated learning (SRL). SRL is a necessary skill for medical students' professional development. However, few medical schools have developed longitudinal, integrated programs of SRL across their clinical curriculum. Methods: Using a program first piloted in our Neurology Clerkship, we developed a system across third-year core clinical clerkships in which students created weekly, patient-centered learning goals that they shared with their attendings and residents. To understand how students and their supervisors used these goals, we conducted semi-structured interviews. Using a constructivist grounded-theory approach, we analyzed the interviews for emerging themes to create a theory-informed model of the factors that influenced students' SRL. Results: Fourteen students and thirteen supervisors were interviewed across all participating clerkships. Saturation was reached within these samples. The factor students described as most significantly influencing the efficacy of their goals was collaborative engagement from their supervisors. Students' own proactive use of the goals and self-advocacy were also important factors in goal efficacy. Students struggled with overload in the busy and random clinical workplace, but many felt that the goals were able to help them focus their attention on certain aspects of their learning. Supervisors described how the goals focused both their efforts as educators and students' work. They noted how lack of student effort could limit the efficacy of the goals. Busy workload limited their ability to engage with students. Conclusions: Our results highlight the significance of collaborative interactions between students and supervisors and the need to overcome challenges of the clinical environment as dominant factors that influence students' SRL. Clerkship directors and clinical faculty must address these factors as they plan and implement programs of SRL.

Purpose Reflection is a key element in learning from experience, but the impact of most programmes of reflection on daily practice remains unclear. We investigated students' perceptions of adding a daily written reflection assignment to a clinical rotation.Methods Third-year medical students on a single two-week rotation completed daily reflections analyzing their performance. Programme evaluation used a 33-question anonymized survey. Quantitative data were summarized and qualitative responses coded for recurring themes.Results Twenty-six students completed the survey (90 % response rate). Eighty-five percent of students felt that the daily reflections had a positive impact on their learning from clinical experience. Seventy-seven percent of students reported that the programme changed their awareness of their thoughts and actions, and 80 % felt that it improved their recall of experiences. A greater sense of mindfulness and focus on self-improvement were major themes that emerge from students' descriptions of the role of daily reflections in their learning.Conclusion Overall, daily reflections demonstrated a positive learning influence. This exploratory study suggests students may benefit from more frequent, short reflections as opposed to more typically spaced reflective assignments.

Background: Self-regulated learning, including student-generated learning goals and flexibility in the learning structure are increasingly being used to enhance medical education. The role of these practices in surgical education of medical students has not been studied.Materials and methods: We administered an 18-question electronic survey to all third-year medical students at Washington University in St. Louis School of Medicine. Of the 126 students invited, 64 responded and 56 were included in the analysis.Results: We found that third-year medical students develop learning goals at the beginning of the surgery clerkship. Although these learning goals theoretically can be a mechanism for enhanced student-faculty engagement, students are not aware of formal mechanisms for sharing these goals with faculty members. Furthermore, students report a lack of flexibility within the surgery clerkship and discomfort with requesting specific learning opportunities. Finally, students report that they believe increased flexibility could improve student engagement, learning, and the overall clerkship experience.Conclusions: We therefore propose that a mechanism for students to share their learning goals with faculty and an infrastructure in which student learning experiences can be tailored to fit with these individualized goals would enhance student surgical learning. (C) 2016 Elsevier Inc. All rights reserved.

Background: Self-regulated learning, including student generated learning goals and flexibility in the learning structure are increasingly being used to enhance medical education. The role of these practices in surgical education of medical students has not been studied. Materials and Methods: We administered an 18 question electronic survey to all third year medical students at Washington University in St. Louis School of Medicine. Of the 126 students invited, 64 responded and 56 were included in the analysis. Results: We found that third year medical students develop learning goals at the beginning of the surgery clerkship. While these learning goals theoretically can be a mechanism for enhanced student-faculty engagement, students are not aware of formal mechanisms for sharing these goals with faculty members. Further, students report a lack of flexibility within the surgery clerkship and discomfort with requesting specific learning opportunities. Finally, students report that they believe increased flexibility could improve student engagement, learning, and the overall clerkship experience. Conclusions: We therefore propose that a mechanism for students to share their learning goals with faculty and an infrastructure in which student learning experiences can be tailored to fit with these individualized goals would enhance student surgical learning.

Introduction A large body of evidence indicates that retrieval practice (test-enhanced learning) and spaced repetition increase long-term information retention. Implementation of these strategies in medical curricula is unfortunately limited. However, students may choose to apply them autonomously when preparing for high-stakes, cumulative assessments, such as the United States Medical Licensing Examination Step 1. Methods Seventy-two medical students at one institution completed a survey concerning their use of user-generated (Anki) or commercially-available (Firecracker) flashcards intended for spaced repetition and of boards-style multiple-choice questions (MCQs). Other information collected included Step 1 score, past academic performance (Medical College Admission Test [MCAT] score, preclinical grades), and psychological factors that may have affected exam preparation or performance (feelings of depression, burnout, and test anxiety). Results All students reported using practice MCQs (mean 3870, SD 1472). Anki and Firecracker users comprised 31 and 49 % of respondents, respectively. In a multivariate regression model, significant independent predictors of Step 1 score included MCQs completed (unstandardized beta coefficient [B] = 2.2 × 10− 3, p < 0.001), unique Anki flashcards seen (B = 5.9 × 10− 4, p = 0.024), second-year honours (B = 1.198, p = 0.002), and MCAT score (B = 1.078, p = 0.003). Conclusions Medical students engage extensively in self-initiated retrieval practice, often with spaced repetition. These practices are associated with superior performance on a medical licensing examination and should be considered for formal support by educators.