Objective: The Electronic-portolio (ePortfolio) has played a major role in postgraduate medical training in the United Kingdom (UK) for many years, having been created to help facilitate lifelong learning. Its use is multifaceted in supporting development through formative and summative assessment, quality assurance and promotion of self-reflection in order to enhance future performance. A recent high profile medicolegal case in the UK illuminated pre-existing issues with the ePortfolio. This study, contacted at the time when the medico legal case was in the appeals process, aimed to establish trainees’ experiences of the ePortfolio, including their perceptions of the trust they held in the system. Methods: As part of a larger study into physician trainees’ perceptions of workplace based assessments value, physician specialty trainees (n=14) attended two focus groups to discuss their perceptions of the ePortfolio process. Grounded Theory methods were applied. Data analysis commenced immediately following collection of the first focus group transcript, in line with Grounded Theory principles (Glaser, 1978). This supported theoretical sampling; allowing the initial data to be used to inform the subsequent steps taken. Line by line coding and memo writing was used throughout, with themes being generated directly out of the data and analysis continuing until theoretical saturation was achieved. Results: Participants identified both benefits and limitations of the ePortfolio. Key concerns related to the permanency of documentation and the potential for negative training, and subsequent career progression implications. The publicised medicolegal case challenged participant trust in the system, with individuals reporting concerns that the ePortfolio suppressed wider organisational issues. Conclusion: Participants identified several factors which appear to impact upon trust of the ePortfolio, which may potentially subvert any benefits associated with its use. Permanency of documentation of suboptimal performance and the identification of inherent biases existing in the ePotfolio appear to be the major driving concerns which threaten optimal engagement with the ePorftolio. The introduction of clearer guidelines for reflective practice and ePortfolio engagement may enhance future trust in the ePortfolio.
Introduction Accreditation Council for Graduate Medical Education's (ACGME's) Milestones assessment requirement has placed new demands on Program Directors (PDs), especially those with limited knowledge of assessment and evaluation activities. There is a lack of clarity on how Program Director (PDs)/Associate PDs (APDs) are effectively implementing milestones assessment and evaluation practices in the Graduate Medical Education programs. The purpose of this study was to investigate current assessment practices, needs, and challenges of PDs in implementing milestones assessment within their residency and fellowship programs in a pediatric hospital setting. Methods This study used a collective case study approach to obtain information from PDs, APDs, and Clinical Competency Committee (CCC) Chairs in 19 graduate programs at a pediatric hospital. We used structured meetings with planned agendas and a pre-formatted template to itemize program needs/difficulties/challenges in the milestone assessment. We used cross-case thematic content analysis to identify categories and themes to compare differences and commonalities across programs. Results A total of 38 PDs, APDs, and CCC Chairs from 19 different specialties/subspecialties participated in this study. Thirteen types of assessment and evaluation tools were consistently used across programs. Three categories emerged in relation to those assessment and evaluation types (direct, indirect, and multi-source). Rotation evaluation (84.2%), direct observation (73.2%), and 360-degree assessment (68.4%) were primarily used for measuring patient care among the six core competencies. Programs’ needs varied from curriculum and assessment tool development to alignment of milestones items, and to creating a systematic assessment management plan. The most common challenges were difficulties related to logistics and tracking of evaluation in the survey management system (52.6%), challenges with time management (47.3%), and difficulty in determining and interpreting the milestones’ numbers and levels (31.5%). Conclusions Milestones assessment and evaluation in medical education can be a challenge, but a priority for many training programs. Our study indicated that milestones assessment and evaluation in medical education are far more complex than we expect. Multiple assessment methods must be utilized to evaluate all essential competencies for accurate measurement of trainees’ performance abilities. Our study uncovered several issues PDs faced during the implementation of milestones assessment and needs and challenges.
Objective: The purpose of this study was to evaluate student achievement and attitudes pertaining to a performance-based assessment in a medical student chest radiology curriculum using mixed quantitative and qualitative methods. Methods: One-hundred-and-sixty-one participating second-year students, in the 2015-16 academic year took a post-curriculum multiple choice question (MCQ) exam. Students also underwent a performance-based assessment in the form of a chest x-ray interpretation small-group session administered by radiology faculty at a picture archiving and communications system (PACS) workstation. Each student verbally interpreted one chest radiograph showing one of 6 pathologies and was given a numerical rating based on a standardized rubric. This score was compared to the correctness of the students answer to the corresponding MCQ question on the same topic. All students completed a post-session questionnaire. Open-ended free-text responses regarding student attitudes were coded into qualitative themes by three independent raters. High inter-rater agreement was demonstrated by average agreement index of .82 or greater (ranging from 0-1, with 1 indicating perfect agreement) for responses to the most frequent themes. Results: There was no significant association between scores on MCQ exam and performance-based assessment. Up to 90% of post-session questionnaire respondents indicated a Likert rating of 5 (strongly agree) when asked if the sessions improved understanding of chest x-rays and their ability to identify specific radiologic pathology. Dominant themes from open-ended responses were then derived. Conclusions: Lack of agreement between student performance on the chest x-ray interpretation small-group session and that on the parallel MCQ exam suggests that they each measure a different type of achievement, with the former emphasizing skills over knowledge. Features most commonly valued by students in the performance-based assessment were its ability to reinforce prior knowledge, supply an authentic and relevant PACS experience simulating real life, and provide an opportunity for active practice of radiology interpretive skills.
Introduction: Healthcare professionals educator development (HPED) is costly, both financially and in terms of clinicians time. However, there is little research into how HPED programmes can be evaluated. Research in other educational fields has demonstrated that the analysis of changes in learners social networks and the associated changes in social capital can reveal interesting and important effects of educational programmes which would otherwise be unknown. However, research on the social network impact of HPED is minimal. Aims: The authors present an innovative exploratory study of a new evaluation methodology, which considers the social network and social capital of a participant after completion of a HPED programme. To the best of the authors knowledge, this is the first time that such a method has been used in HPED. Method: Mixed methods social network analysis (MMSNA) was used to measure and further understand the social capital of the participant after completion of a HPED programme. Data collection was via a self-report template and a semi-structured interview. An exploratory case study of a medical doctor who completed a HPED programme at masters level was conducted in October 2018 at Edge Hill University, Ormskirk, England. Results: Relationships made through the HPED programme provided the participant with access to social capital in the form of educational expertise, knowledge and information about job opportunities. These new relationships changed the social network structure, with reduced network constraint and increased number of structural holes in the network of the participant. Such access to resources unavailable to others within the network placed the participant at an ongoing advantage. Conclusion: MMSNA can reveal key benefits of HPED programmes which would not be apparent with other methodologies. The methodology produces results that can be transferred to other HPED programmes.
Seasonal variation to manuscript submission and publication in the medical literature is yet to be carefully scrutinized. We report a lack of significant seasonal bias in relation to medical student submission to a prominent Australasian medical journal. Factors that contribute to when a manuscript is submitted may be predictable. Factors that contribute to when a manuscript is published, on the other hand, appear to be less predictable. These observations remain preliminary as the studies examining seasonal bias in the medical literature are too few and methodologically heterogeneous. A careful consideration of the implications will need to be taken into account to ensure a fair peer-review process.
