INTRODUCTION:Targeted identification, effective triage, and rapid hemorrhage control are essential for optimal outcomes of mass-casualty incidents (MCIs). An important aspect of Emergency Medical Service (EMS) care is field triage, but this skill is difficult to teach, assess, and research. STUDY OBJECTIVE:This study assessed triage efficacy and hemorrhage control of emergency responders from different professions who used the Sort, Assess, Life-Saving Treatment (SALT) triage algorithm in a virtual reality (VR) simulation of a terrorist subway bombing. METHODS:After a brief just-in-time training session on the SALT triage algorithm, participants applied this learning in First VResponder, a high-fidelity VR simulator (Tactical Triage Technologies, LLC; Powell, Ohio USA). Participants encountered eleven virtual patients in a virtual scene of a subway station that had experienced an explosion. Patients represented individuals with injuries of varying severity. Metrics assessed included triage accuracy and treatment efficiency, including time to control life-threatening hemorrhage. Independent Mann-Whitney analyses were used to compare two professional groups on key performance variables. RESULTS:The study assessed 282 participants from the ranks of EMS clinicians and medical trainees. Most (94%) participants correctly executed both global SALT sort commands. Participants triaged and treated the entire scene in a mean time of 7.8 decimal minutes, (95%CI, 7.6-8.1; SD = 1.9 decimal minutes) with a patient triage accuracy rate of 75.8% (95%CI, 74.0-77.6; SD = 15.0%). Approximately three-quarters (77%) of participants successfully controlled all life-threatening hemorrhage, within a mean time of 5.3 decimal minutes (95%CI, 5.1-5.5; SD = 1.7 decimal minutes). Mean time to hemorrhage control per patient was 0.349 decimal minutes (SD = 0.349 decimal minutes). Overall, EMS clinicians were more accurate with triage (P ≤ .001) and were faster at triage, total hemorrhage control (P < .01), and hemorrhage control per patient (P < .004) than medical trainees. CONCLUSIONS:Through assessments using VR simulation, it was observed that more experienced individuals from the paramedic (PM) workforce out-performed less experienced medical trainees. The study also observed that the medical trainees performed acceptably, even though their only formal training in SALT triage was a 30-minute, just-in-time lecture. Both of these findings are important for establishing evidence that VR can serve as a valid platform for assessing the complex skills of triage and treatment of an MCI, including the assessment of rapid hemorrhage control.

OBJECTIVE:This study explored the prevalence and attributes of triage errors made by emergency responders during virtual reality simulations of mass casualty incidents. METHODS:The study analyzed errors made by 99 emergency responders during their triage and treatment of a mass casualty incident in virtual reality. Responders received training on the Sort, Assess, Life-saving Intervention, Treatment, Transport (SALT) protocol, then responded to a virtual bombed subway station. Responder accuracy, efficiency, and application of treatments were tracked. Error analysis was performed through the lens of human factors. Accordingly, errors were categorized by their nature: either perception, proficiency, or procedure. RESULTS:Responders correctly triaged 70% of virtual patients, and 78% demonstrated relative efficiency. Interaction times between responders and patients averaged 20 seconds. The time to assess and treat all patients for life-threatening bleeding injuries across the entire scene averaged six minutes. Most errors were related to proficiency (e.g., competence or experience). However, procedural errors (shortcomings of SALT) and perceptual errors (degraded sensory input from programmed environmental chaos, i.e., virtual smoke/debris and louder sound) were also observed. Most errors were related to patients with either respiratory issues or multiple injuries. CONCLUSION:Virtual reality (VR) offered a controlled environment for studying errors made by emergency responders in a mass casualty incident, which will lead to improved training and protocols to better prepare them for these events.

SALT (sort-assess-lifesaving interventions-treatment and/or transport) was developed as a national all-hazards mass casualty initial triage standard for all patients (eg, adults, children, and special populations).1 SALT is endorsed by1-3 the American College of Emergency Physicians (ACEP), American College of Surgeons Committee on Trauma, American Trauma Society, National Association of EMS Physicians (NAEMSP), and National Disaster Life Support Education Consortium. The process is explained using Video 1, and its algorithm is presented in Figure 1. Hemorrhage is the leading preventable cause of death in trauma (30%–40% of fatalities).4 Death from hemorrhage can occur in as little as 3–5 minutes. SALT allows for early hemorrhage control. In other triage algorithms, bleeding control does not occur until after respirations are counted and perfusion (cap refill) is checked. to control major hemorrhage with tourniquets or direct pressure provided by other patients or other devices; to open the airway through positioning or basic airway adjuncts (no advanced airway devices should be used); if the patient is a child, consider giving two rescue breaths; chest needle decompression; CBRN antidotes or autoinjector antidotes: SALT includes an expectant category for patients who are still breathing but unlikely to survive given current resources. SALT allows for a quick global sorting of the patients so responders can more accurately identify patients who are not responding or have an obvious life threat. VR SALT demonstration is presented using Video 2.6 This work was supported through a grant from the United States Agency for Healthcare Research and Quality Grant Number R18HS025915. The authors report no conflicts of interest.

Abstract Background To minimize loss of life, modern mass casualty response requires swift identification, efficient triage categorization, and rapid hemorrhage control. Current training methods remain suboptimal. Our objective was to train first responders to triage a mass casualty incident using Virtual Reality (VR) simulation and obtain their impressions of the training’s quality and effectiveness. We trained subjects in a triage protocol called Sort, Assess, Lifesaving interventions, and Treatment and/or Transport (SALT) Triage then had them respond to a terrorist bombing of a subway station using a fully immersive virtual reality simulation. We gathered learner reactions to their virtual reality experience and post-encounter debriefing with a custom electronic survey. The survey was designed to gather information about participants’ demographics and prior experience, including roles, triage training, and virtual reality experience. We then asked them to evaluate the training and encounter and the system’s potential for training others. Results We received 375 completed evaluation surveys from subjects who experienced the virtual reality encounter. Subjects were primarily paramedics, but also included medical learners as well as other emergency medical service (EMS) professionals. Most participants (95%) recommended the experience for other first responders and rated the simulation (95%) and virtual patients (91%) as realistic. Ninety-four percent (94%) of participants rated the virtual reality simulator as “excellent” or “good.” We observed some differences between emergency medical service and medical professionals regarding their prior experience with disaster response training and their opinions on how much the experience contributed to their learning. We observed no differences between subjects with extensive virtual reality experience and those without. Conclusions Our virtual reality simulator is an automated, customizable, fully immersive virtual reality system for training and assessing personnel in the proper response to a mass casualty incident. Participants perceived the simulator as an adequate alternative to traditional triage and treatment training and believed that the simulator was realistic and effective for training. Prior experience with virtual reality was not a prerequisite for the use of this system.

Objective: Hemorrhage control, triage efficiency, and triage accuracy are essential skills for optimal outcomes in mass casualty incidents. This study evaluated user application of skills through a Virtual Reality (VR) simulation of a subway bombing. Methods: EMS clinicians and healthcare professionals engaged in a VR simulation of a bomb/ blast scenario utilizing VRFirstResponder, a high-fidelity, fully immersive, automated, customizable, and programmable VR simulation platform. Metrics including time to control life- threatening hemorrhage and triage efficacy were analyzed using median and interquartile ranges (IQR). Results: 389 EMS responders engaged in this high-fidelity VR simulation encountering 11 virtual patients with varying injury severity. The median time to triage the scene was 7:38 minutes (SD = 2:27, IQR = 6:13, 8:59). A robust 93% of participants successfully implemented all required hemorrhage control, with a median time of 3:51 minutes for life-threatening hemorrhage control (SD = 1:44, IQR = 2:41, 4:52). Hemorrhage control per patient took a median of 11 seconds (SD = 0:47, IQR = 0:06, 0:20). Participants accurately tagged 73% of patients and 17% effectively utilized the SALT sort commands for optimal patient evaluation. Conclusion: The VRFirstResponder simulation, currently under validation, aims to enhance realism by incorporating distractors and refining assessment tools.

Background Medical students in the U.S. must demonstrate urgent and emergent care competence before graduation. Urgent and emergent care competence involves recognizing, evaluating and initiating management of an unstable patient. High-fidelity (HF) simulation can improve urgent and emergent care skills, but because it is resource intense, alternative methods are needed.Study Objective Our primary purpose was to use program evaluations to compare medical student experiences with HF and virtual reality (VR) simulations as assessment platforms for urgent and emergent care skills.Methods During their emergency medicine clerkship, students at The Ohio State University College of Medicine must demonstrate on HF manikins, competence in recognizing and initiating care of a patient requiring urgent or emergent care. Students evaluated these simulations on a five-point quality scale and answered open-ended questions about simulation strengths and weaknesses. Faculty provided feedback on student competence in delivering urgent or emergent care. In 2022, we introduced VR as an alternative assessment platform. We used Wilcoxon Signed Ranks and Boxplots to compare ratings of HF to VR and McNemar Test to compare competence ratings. Comments were analyzed with summative content analysis or thematic coding.Results We received at least one evaluation survey from 160 of 216 (74.1%) emergency medicine clerkship students. We were able to match 125 of 216 (57.9%) evaluation surveys for students who completed both. Average ratings of HF simulations were 4.6 of 5, while ratings of VR simulations were slightly lower at 4.4. Comments suggested that feedback from both simulation platforms was valued. Students described VR as novel, immersive, and good preparation for clinical practice. Constructive criticism identified the need for additional practice in the VR environment. Student performance between platforms was significantly different with 91.7% of students achieving competence in HF, but only 65.5% in VR (p≤.001, odds-ratio = 5.75).Conclusion VR simulation functions similarly to HF for formative assessment of urgent and emergent care competence. However, using VR simulation for summative assessment of urgent and emergent care competence must be considered with caution because students require considerable practice and acclimation to the virtual environment.

Abstract Objective To minimize loss of life, mass casualty response requires swift identification, efficient triage categorization, and rapid hemorrhage control. Current training methods remain suboptimal. Our objective was to train first responders to triage a mass casualty incident using Virtual Reality (VR) simulation and obtain their impressions of the training’s quality and effectiveness. Methods We trained subjects in SALT Triage then had them respond to a terrorist bombing of a subway station using a fully immersive VR simulation. We gathered learner reactions to their VR experience and post-encounter debriefing with a custom electronic survey. Results Nearly 400 subjects experienced the VR encounter and completed evaluation surveys. Most participants (95%) recommended the experience for other first responders and rated the simulation (95%) and virtual patients (91%) as realistic. Ninety-four percent of participants rated the VR simulator as “excellent” or “good.” We observed no differences between those who owned a personal VR system and those who did not. Conclusions Our VR simulator (go.osu.edu/firstresponder) is an automated, customizable, fully immersive virtual reality system for training and assessing personnel in the proper response to a mass casualty incident. Participants perceived the encounter as effective for training, regardless of their prior experience with virtual reality.

Randomized prospective studies represent the gold standard for experimental design. In this paper, we present a randomized prospective study to validate the benefits of combining rule-based and data-driven natural language understanding methods in a virtual patient dialogue system. The system uses a rule-based pattern matching approach together with a machine learning (ML) approach in the form of a text-based convolutional neural network, combining the two methods with a simple logistic regression model to choose between their predictions for each dialogue turn. In an earlier, retrospective study, the hybrid system yielded a nearly 50% error reduction on our initial data, in part due to the differential performance between the two methods as a function of label frequency. Given these gains, and considering that our hybrid approach is unique among virtual patient systems, we compare the hybrid system to the rule-based system by itself in a randomized prospective study. We evaluate 110 unique medical student subjects interacting with the system over 5,296 conversation turns, to verify whether similar gains are observed in a deployed system. This prospective study broadly confirms the findings from the earlier one but also highlights important deficits in our training data. The hybrid approach still improves over either rule-based or ML approaches individually, even handling unseen classes with some success. However, we observe that live subjects ask more out-of-scope questions than expected. To better handle such questions, we investigate several modifications to the system combination component. These show significant overall accuracy improvements and modest F1 improvements on out-of-scope queries in an offline evaluation. We provide further analysis to characterize the difficulty of the out-of-scope problem that we have identified, as well as to suggest future improvements over the baseline we establish here.

As mass casualty incidents continue to escalate in the United States, we must improve frontline responder performance to increase the odds of victim survival. In this article, we describe the First Responder Virtual Reality Simulator, a high-fidelity, fully immersive, automated, programmable virtual reality (VR) simulation designed to train frontline responders to treat and triage victims of mass casualty incidents. First responder trainees don a wireless VR head-mounted display linked to a compatible desktop computer. Trainees see and hear autonomous, interactive victims who are programmed to simulate individuals with injuries consistent with an explosion in an underground space. Armed with a virtual medical kit, responders are tasked with triaging and treating the victims on the scene. The VR environment can be made more challenging by increasing the environmental chaos, adding patients, or increasing the acuity of patient injuries. The VR platform tracks and records their performance as they navigate the disaster scene. Output from the system provides feedback to participants on their performance. Eventually, we hope that the First Responder system will serve both as an effective replacement for expensive conventional training methods as well as a safe and efficient platform for research on current triage protocols.

INTRODUCTION:Advances in natural language understanding have facilitated the development of Virtual Standardized Patients (VSPs) that may soon rival human patients in conversational ability. We describe herein the development of an artificial intelligence (AI) system for VSPs enabling students to practice their history taking skills.METHODS:Our system consists of (1) Automated Speech Recognition (ASR), (2) hybrid AI for question identification, (3) classifier to choose between the two systems, and (4) automated speech generation. We analyzed the accuracy of the ASR, the two AI systems, the classifier, and student feedback with 620 first year medical students from 2018 to 2021.RESULTS:System accuracy improved from ∼75% in 2018 to ∼90% in 2021 as refinements in algorithms and additional training data were utilized. Student feedback was positive, and most students felt that practicing with the VSPs was a worthwhile experience.CONCLUSION:We have developed a novel hybrid dialogue system that enables artificially intelligent VSPs to correctly answer student questions at levels comparable with human SPs. This system allows trainees to practice and refine their history-taking skills before interacting with human patients.

BackgroundEntrustable Professional Activities (EPAs) refers to discrete clinical activities that requires the utilization and integration of various competencies. EPA-10 requires a physician to recognize an unstable patient and initiate evaluation and management. Research has demonstrated that fourth-year medical students are less prepared to manage unstable patients as compared to nonacute conditions. Previous research has demonstrated that simulation-based medical education can improve the acute care skills of medical students and residents. However, high-fidelity (HF) simulation requires expensive equipment and qualified technical support staff. An alternative platform for training and assessing is virtual reality (VR).Study Objective: As an emerging technology, relatively little research has been published on either the utility or cost effectiveness of VR simulation in medical education. Moreover, no research to date has attempted to study VR simulation assessments of entrustable professional activities in medical students. The project attempts to compare the cost and effectiveness of VR simulation directly with HF simulation as an assessment platform for EPA-10.MethodsAs part of the Emergency Medicine Clerkship at The Ohio State University, all fourth year students participate in a HF simulation performance assessment of an emergent patient (EPA-10). Two of the EPA-10 cases were built on the VR simulation platform SimX and a virtual reality EPA-10 simulation was incorporated into the clerkship. A total of 171 fourth-year medical students were able to participate in both the HF and VR simulated cases. Performance data, cost data, and participant perceptions were collected for both the VR and HF simulations throughout the academic year.ResultsWhen participants were asked to "Rate the overall quality of this session", the majority of the medical students rated the quality of the VR session either "Very Good" and "Excellent" (M = 3.57, SD = .75). Similarly, the majority of the medical students rated the quality of the HF session either "Very Good" and "Excellent" (M = 3.66, SD = .64). Subjective participant comments included the perceived utility of VR simulation as compared to other educational activities in the clerkship and requests for additional sessions. Constructive criticism focused on the need for a more detailed orientation to the simulated environment. In addition, faculty were asked to assess the performance of the student participants using the question, "Would you feel confident in this student's ability to manage an acutely decompensating/acutely ill patient with a life threatening illness?" In the VR simulations, 32.7% of students did not meet entrustment, while in the HF simulations, 8.6% of students did not meet entrustment. The difference between these entrustment rates was thought to be multifactorial, but may partially be the result of the VR gameplay learning curve. The cost of the hardware, software licensure and two custom cases was approximately $13,000.ConclusionVirtual reality simulation can be used as an effective formative assessment tool for EPA-10 and can function similarly to high-fidelity simulation in this role. Use of virtual reality simulation as a summative assessment of EPA-10 can be considered with caution, as multiple factors can contribute to participant performance and entrustment rates.No, authors do not have interests to disclose BackgroundEntrustable Professional Activities (EPAs) refers to discrete clinical activities that requires the utilization and integration of various competencies. EPA-10 requires a physician to recognize an unstable patient and initiate evaluation and management. Research has demonstrated that fourth-year medical students are less prepared to manage unstable patients as compared to nonacute conditions. Previous research has demonstrated that simulation-based medical education can improve the acute care skills of medical students and residents. However, high-fidelity (HF) simulation requires expensive equipment and qualified technical support staff. An alternative platform for training and assessing is virtual reality (VR).Study Objective: As an emerging technology, relatively little research has been published on either the utility or cost effectiveness of VR simulation in medical education. Moreover, no research to date has attempted to study VR simulation assessments of entrustable professional activities in medical students. The project attempts to compare the cost and effectiveness of VR simulation directly with HF simulation as an assessment platform for EPA-10. Entrustable Professional Activities (EPAs) refers to discrete clinical activities that requires the utilization and integration of various competencies. EPA-10 requires a physician to recognize an unstable patient and initiate evaluation and management. Research has demonstrated that fourth-year medical students are less prepared to manage unstable patients as compared to nonacute conditions. Previous research has demonstrated that simulation-based medical education can improve the acute care skills of medical students and residents. However, high-fidelity (HF) simulation requires expensive equipment and qualified technical support staff. An alternative platform for training and assessing is virtual reality (VR). Study Objective: As an emerging technology, relatively little research has been published on either the utility or cost effectiveness of VR simulation in medical education. Moreover, no research to date has attempted to study VR simulation assessments of entrustable professional activities in medical students. The project attempts to compare the cost and effectiveness of VR simulation directly with HF simulation as an assessment platform for EPA-10. MethodsAs part of the Emergency Medicine Clerkship at The Ohio State University, all fourth year students participate in a HF simulation performance assessment of an emergent patient (EPA-10). Two of the EPA-10 cases were built on the VR simulation platform SimX and a virtual reality EPA-10 simulation was incorporated into the clerkship. A total of 171 fourth-year medical students were able to participate in both the HF and VR simulated cases. Performance data, cost data, and participant perceptions were collected for both the VR and HF simulations throughout the academic year. As part of the Emergency Medicine Clerkship at The Ohio State University, all fourth year students participate in a HF simulation performance assessment of an emergent patient (EPA-10). Two of the EPA-10 cases were built on the VR simulation platform SimX and a virtual reality EPA-10 simulation was incorporated into the clerkship. A total of 171 fourth-year medical students were able to participate in both the HF and VR simulated cases. Performance data, cost data, and participant perceptions were collected for both the VR and HF simulations throughout the academic year. ResultsWhen participants were asked to "Rate the overall quality of this session", the majority of the medical students rated the quality of the VR session either "Very Good" and "Excellent" (M = 3.57, SD = .75). Similarly, the majority of the medical students rated the quality of the HF session either "Very Good" and "Excellent" (M = 3.66, SD = .64). Subjective participant comments included the perceived utility of VR simulation as compared to other educational activities in the clerkship and requests for additional sessions. Constructive criticism focused on the need for a more detailed orientation to the simulated environment. In addition, faculty were asked to assess the performance of the student participants using the question, "Would you feel confident in this student's ability to manage an acutely decompensating/acutely ill patient with a life threatening illness?" In the VR simulations, 32.7% of students did not meet entrustment, while in the HF simulations, 8.6% of students did not meet entrustment. The difference between these entrustment rates was thought to be multifactorial, but may partially be the result of the VR gameplay learning curve. The cost of the hardware, software licensure and two custom cases was approximately $13,000. When participants were asked to "Rate the overall quality of this session", the majority of the medical students rated the quality of the VR session either "Very Good" and "Excellent" (M = 3.57, SD = .75). Similarly, the majority of the medical students rated the quality of the HF session either "Very Good" and "Excellent" (M = 3.66, SD = .64). Subjective participant comments included the perceived utility of VR simulation as compared to other educational activities in the clerkship and requests for additional sessions. Constructive criticism focused on the need for a more detailed orientation to the simulated environment. In addition, faculty were asked to assess the performance of the student participants using the question, "Would you feel confident in this student's ability to manage an acutely decompensating/acutely ill patient with a life threatening illness?" In the VR simulations, 32.7% of students did not meet entrustment, while in the HF simulations, 8.6% of students did not meet entrustment. The difference between these entrustment rates was thought to be multifactorial, but may partially be the result of the VR gameplay learning curve. The cost of the hardware, software licensure and two custom cases was approximately $13,000. ConclusionVirtual reality simulation can be used as an effective formative assessment tool for EPA-10 and can function similarly to high-fidelity simulation in this role. Use of virtual reality simulation as a summative assessment of EPA-10 can be considered with caution, as multiple factors can contribute to participant performance and entrustment rates.No, authors do not have interests to disclose Virtual reality simulation can be used as an effective formative assessment tool for EPA-10 and can function similarly to high-fidelity simulation in this role. Use of virtual reality simulation as a summative assessment of EPA-10 can be considered with caution, as multiple factors can contribute to participant performance and entrustment rates.

Objective To provide preclinical medical students early access to otolaryngologists to learn about the specialty, facilitate acquisition of clinical skills, and provide one-on-one mentorship. Methods Students are matched with a single otolaryngology faculty mentor from The Ohio State University/Nationwide Children’s Hospital and attend 8 hours per month in the clinic or operating room, monthly lectures, and rounds, and they give a final presentation. Mentors complete performance evaluations, and surveys are administered longitudinally until Match Day. Results Thirty-five students and 17 faculty members have participated in the program since 2015. All mentors and students found the program to be a valuable experience. When compared to nonparticipating students, participants had significantly higher confidence scores for clinical performance, knowledge of anatomy, and familiarity with the department of interest. All students felt the program prepared them well for third and fourth years, and all 8 of the initial program participants successfully matched into residency with 4 entering otolaryngology. Discussion Medical students face a competitive residency application process in otolaryngology with limited exposure, which creates an opportunity for guidance in the pursuit of matching into this field. This novel preclinical mentorship program prepares students for their clinical years and residency by facilitating acquisition of various competencies. Students gain hands-on clinical exposure in a field of interest and support for navigating the application process. Implications for Practice The structure of this program can be applied to other medical schools or specialties if the individual departments contain adequate resources of teaching faculty willing to participate.

Introduction: Practicing a medical history using standardized patients is an essential component of medical school curricula. Recent advances in technology now allow for newer approaches for practicing and assessing communication skills. We describe herein a virtual standardized patient (VSP) system that allows students to practice their history taking skills and receive immediate feedback. Methods: Our VSPs consist of artificially intelligent, emotionally responsive 3D characters which communicate with students using natural language. The system categorizes the input questions according to specific domains and summarizes the encounter. Automated assessment by the computer was compared to manual assessment by trained raters to assess accuracy of the grading system. Results: Twenty dialogs chosen randomly from 102 total encounters were analyzed by three human and one computer rater. Overall scores calculated by the computer were not different than those provided by the human raters, and overall accuracy of the computer system was 87%, compared with 90% for human raters. Inter-rater reliability was high across 19 of 21 categories. Conclusions: We have developed a virtual standardized patient system that can understand, respond, categorize, and assess student performance in gathering information during a typical medical history, thus enabling students to practice their history-taking skills and receive immediate feedback.

Background: Prenatal counseling at the limits of newborn viability involves sensitive interactions between neonatal providers and families. Empathetic discussions are currently learned through practice in times of high stress. Decision aids may help improve provider communication but have not been universally adopted. Virtual standardized patients are increasingly recognized as a modality for education, but prenatal counseling simulations have not been described. To be valuable as a tool, a virtual patient would need to accurately portray emotions and elicit a realistic response from the provider. Objective: To determine if neonatal providers can accurately identify a standardized virtual prenatal patient's emotional states and examine the frequency of empathic responses to statements made by the patient. Methods: A panel of Neonatologists, Simulation Specialists, and Ethicists developed a dialogue and identified empathic responses. Virtual Antenatal Encounter and Standardized Simulation Assessment (VANESSA), a screen-based simulation of a woman at 23 weeks gestation, was capable of displaying anger, fear, sadness, and happiness through animations. Twenty-four neonatal providers, including a subgroup with an ethics interest, were asked to identify VANESSA's emotions 28 times, respond to statements, and answer open-ended questions. The emotions were displayed in different formats: without dialogue, with text dialogue, and with audio dialogue. Participants completed a post-encounter survey describing demographics and experience. Data were reported using descriptive statistics. Qualitative data from open ended questions (eg, "What would you do?") were examined using thematic analysis. Results: Half of our participants had over 10 years of clinical experience. Most participants reported using medical research (18/23, 78%) and mortality calculators (17/23, 74%). Only the ethics-interested subgroup (10/23, 43%) listed counseling literature (7/10, 70%). Of 672 attempts, participants accurately identified VANESSA's emotions 77.8% (523/672) of the time, and most (14/23, 61%) reported that they were confident in identifying these emotions. The ethics interest group was more likely to choose empathic responses (P=.002). Participants rated VANESSA as easy to use (22/23, 96%) and reported that she had realistic dialogue (15/23, 65%). Conclusions: This pilot study shows that a prenatal counseling simulation is feasible and can yield useful data on prenatal counseling communication. Our participants showed a high rate of emotion recognition and empathy in their responses.

When interpreting questions in a virtual patient dialogue system one must inevitably tackle the challenge of a long tail of relatively infrequently asked questions. To make progress on this challenge, we investigate the use of paraphrasing for data augmentation and neural memory-based classification, finding that the two methods work best in combination. In particular, we find that the neural memory-based approach not only outperforms a straight CNN classifier on low frequency questions, but also takes better advantage of the augmented data created by paraphrasing, together yielding a nearly 10% absolute improvement in accuracy on the least frequently asked questions.

For medical students, virtual patient dialogue systems can provide useful training opportunities without the cost of employing actors to portray standardized patients. This work utilizes word- and character-based convolutional neural networks (CNNs) for question identification in a virtual patient dialogue system, outperforming a strong word- and character-based logistic regression baseline. While the CNNs perform well given sufficient training data, the best system performance is ultimately achieved by combining CNNs with a hand-crafted pattern matching system that is robust to label sparsity, providing a 10% boost in system accuracy and an error reduction of 47% as compared to the pattern-matching system alone.

Immersive learning environments that use virtual simulation (VS) technology are increasingly relevant as medical learners train in an environment of restricted clinical training hours and a heightened focus on patient safety. We conducted a consensus process with a breakout group of the 2017 Academic Emergency Medicine Consensus Conference "Catalyzing System Change Through Health Care Simulation: Systems, Competency, and Outcomes." This group examined the current uses of VS in training and assessment, including limitations and challenges in implementing VS into medical education curricula. We discuss the role of virtual environments in formative and summative assessment. Finally, we offer recommended areas of focus for future research examining VS technology for assessment, including high-stakes assessment in medical education. Specifically, we discuss needs for determination of areas of focus for VS training and assessment, development and exploration of virtual platforms, automated feedback within such platforms, and evaluation of effectiveness and validity of VS education.

Introduction Although traditional virtual patient simulations are designed to teach and assess clinical reasoning skills, few employ conversational dialogue with the patients. The virtual standardized patients (VSPs) described herein represent standardized patients that students interview using natural language. Students take histories and develop differential diagnoses of the VSPs as much as they would with standardized or actual patients. The student-VSP interactions are recorded, creating a comprehensive record of questions and the order in which they were asked, which can be analyzed to assess information-gathering skills. Students document the encounter in an electronic medical record created for the VSPs. Methods The VSP was developed by integrating a dialogue management system (ChatScript) with emotionally responsive 3D characters created in a high-fidelity game engine (Unity). The system was tested with medical students at the Ohio State University College of Medicine. Students are able to take a history of a VSP, develop a differential diagnosis, and document the encounter in the electronic medical record. Results Accuracy of the VSP responses ranged from 79% to 86%, depending on the complexity of the case, type of history obtained, and skill of the student. Students were able to accurately develop an appropriate differential diagnosis on the basis of the information provided by the patient during the encounter. Conclusions The VSP enables students to practice their history-taking skills before encounters with standardized or actual patients. Future developments will focus on creating an assessment module that will automatically analyze VSP sessions and provide immediate student feedback.

We present a corpus of virtual patient dialogues to which we have added manually annotated gold standard word alignments. Since each question asked by a medical student in the dialogues is mapped to a canonical, anticipated version of the question, the corpus implicitly defines a large set of paraphrase (and non-paraphrase) pairs. We also present a novel process for selecting the most useful data to annotate with word alignments and for ensuring consistent paraphrase status decisions. In support of this process, we have enhanced the earlier Edinburgh alignment tool (Cohn et al., 2008) and revised and extended the Edinburgh guidelines, in particular adding guidance intended to ensure that the word alignments are consistent with the overall paraphrase status decision. The finished corpus and the enhanced alignment tool are made freely available.

Evan Jaffe, Michael White, William Schuler, Eric Fosler-Lussier, Alex Rosenfeld, Douglas Danforth. Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications. 2015.