Background: Patients newly diagnosed with diabetes mellitus (diabetes), who require insulin must acquire diabetes "survival" skills prior to discharge home. COVID-19 revealed considerable limitations of traditional in- person, time-intensive delivery of diabetes education and survival skills training (diabetes survival skills training). Furthermore, diabetes survival skills training has not been designed to meet the specific learning needs of patients with diabetes and their caregivers, particularly if delivered by telehealth. The objective of the study was to identify and understand the needs of users (patients newly prescribed insulin and their caregivers) to inform the design of a diabetes survival skills training, specifically for telehealth delivery, through the application of user-centered design and adult learning and education principles. Methods: Users included patients newly prescribed insulin, their caregivers, and laypersons without diabetes. In semi-structured interviews, users were asked about experienced or perceived challenges in learning diabetes survival skills. Interviews were audio-recorded and transcribed. Investigators performed iterative rounds of coding of interview transcripts utilizing a constant comparative method to identify themes describing the dominant challenges users experienced. Themes were then mapped to adult learning and education principles to identify novel educational design solutions that can be applied to telehealth-based learning. Results: We interviewed 18 users: patients (N = 6, 33 %), caregivers (N = 4, 22 %), and laypersons (N = 8, 44 %). Users consistently described challenges in understanding diabetes survival skills while hospitalized; in preparing needed supplies to execute diabetes survival skills; and in executing diabetes survival skills at home. The challenges mapped to three educational strategies: (1) spiral learning; (2) repetitive goal directed practice and feedback, which have the potential to translate into design solutions supporting remote/virtual learning; and (3) form fits function organizer, which supports safe organization and use of supplies to execute diabetes survival skills independently. Conclusion: Learning complex tasks, such as diabetes survival skills, requires time, repetition, and continued support. The combination of a user-centered design approach to uncover learning needs as well as identification of relevant adult learning and education principles could inform the design of more user-centered, feasible, effective, and sustainable diabetes survival skills training for telehealth delivery.

Abstract Disclosure: S.J. Freeman: None. B. Radonski: None. L. Lecka: Employee; Self; Doximity. Stock Owner; Self; Doximity. K. Davis: None. G. Prince: None. K. Carthy: None. J.J. Seley: Speaker; Self; Lifescan Diabetes Institute. J. Song: None. J. Lee: None. S.C. Bailey: Consulting Fee; Self; Merck, Lundbeck, Sanofi-Aventis, Pfizer, Inc., Luto, University of Westminster, Gilead. Grant Recipient; Self; Merck, Eli Lilly & Company, Pfizer, Inc., Lundbeck, Gordon and Betty Moore Foundation, National Institutes of Health, Gilead. R. Khorzad: None. D. Gatchell: None. B. Ankenman: None. D.R. Lewis: Grant Recipient; Self; Pfizer, Inc., Spencer Foundation, National Institutes of Health. J. Holl: None. A. Wallia: Consulting Fee; Self; Eli Lilly & Company. Grant Recipient; Self; Novo Nordisk. Research Investigator; Self; UnitedHealth Group, Eli Lilly & Company. Patient-centered approaches for teaching diabetes mellitus (DM) survival skills are essential. Furthermore, in the peri-COVID era, interventions also need to be amenable to remote care delivery. User-Centered design (UCD) including usability testing is a key strategy to optimize adoption and engagement of interventions. We developed a Diabetes Survival Skills Toolkit (website, paper guide, and a physical Kit with simulation supplies) using UCD (> 50 sessions), followed by administration of system usability surveys (SUS) (scored as unacceptable, acceptable, or excellent) and, in a subset, additional skills testing. Skills testing included simulated blood glucose checks and insulin administration, conducted by 2 trained observers. Forty-three participants with no prior history of DM were recruited between 01/2021-07/2022 to independently learn survival skills using different Toolkit components [website only (N=11), Kit + paper guide (N=28), and Kit + website (N=4)]. Purposive sampling for age and highest education level resulted in 33% being ≥ 65 years and 35% having < 4-year degree. Overall, SUS scores were deemed excellent (N=15/43 [35%]) or acceptable (N=20/43 [47%]). Unacceptable scores were noted in 8/43 (19%) [4 website only (all > 4-year degree) and 4 Kit + paper guide (3 of 4 > 65 years, all < 4-year degree)]. Use of the website alone resulted in a higher rate of unacceptable SUS scores (37%) compared to use of the Kit with either the paper guide or website (13%). SUS-score category was not associated with age (82% acceptable/excellent among <45 years, 86% among 45-64 years, and 79% among >=65 years; Fishers’ p=1.00) nor highest education level (80% acceptable/excellent among <4-year degree and 82% among >=4-year degree; Fisher’s p=0.69). Participants who completed skills testing (N= 28 Kit + paper guide, 4 Kit + website), regardless of their SUS score, all correctly demonstrated the ability to inject insulin with simulation supplies. However, 4/32 (13%) (all SUS scores acceptable/excellent) were unable to navigate all steps independently and 9/32 (28%) (2 SUS unacceptable) did not use the recommended instructional pathway. All 4 participants (3 with > age 65 and < 4-year degree) who completed skills testing but had unacceptable SUS scores still correctly demonstrated the ability to measure blood glucose and inject insulin. In conclusion, a Survival Skills Toolkit, resulted in excellent rates of successful survival skills performance when tested with laypersons of diverse ages and education levels. Subjective usability (SUS scores) did differ among users of different Toolkit components; however, they did not align with actual skill performance. Design preferences and usability tests as well as subsequent skills testing are critical to optimally design tools for diabetes survival skills training. Presentation: Saturday, June 17, 2023

Computer experiments are often used as inexpensive alternatives to real‐world experiments. Statistical metamodels of the computer model's input‐output behavior can be constructed to serve as approximations of the response surface of the real‐world system. The suitability of a metamodel depends in part on its intended use. While decision makers may want to understand the entire response surface, they may be particularly keen on finding interesting regions of the design space, such as where the gradient is steep. We present an adaptive, value‐enhanced batch sequential algorithm that samples more heavily in such areas while still providing an understanding of the entire surface. The design points within each batch can be run in parallel to leverage modern multi‐core computing assets. We illustrate our approach for deterministic computer models, but it has potential for stochastic simulation models as well.

Background and Purpose:Accurate prehospital diagnosis of stroke by emergency medical services (EMS) can increase treatments rates, mitigate disability, and reduce stroke deaths. We aimed to develop a model that utilizes natural language processing of EMS reports and machine learning to improve prehospital stroke identification.Methods:We conducted a retrospective study of patients transported by the Chicago EMS to 17 regional primary and comprehensive stroke centers. Patients who were suspected of stroke by the EMS or had hospital-diagnosed stroke were included in our cohort. Text within EMS reports were converted to unigram features, which were given as input to a support-vector machine classifier that was trained on 70% of the cohort and tested on the remaining 30%. Outcomes included final diagnosis of stroke versus nonstroke, large vessel occlusion, severe stroke (National Institutes of Health Stroke Scale score >5), and comprehensive stroke center-eligible stroke (large vessel occlusion or hemorrhagic stroke).Results:Of 965 patients, 580 (60%) had confirmed acute stroke. In a test set of 289 patients, the text-based model predicted stroke nominally better than models based on the Cincinnati Prehospital Stroke Scale (c-statistic: 0.73 versus 0.67, P=0.165) and was superior to the 3-Item Stroke Scale (c-statistic: 0.73 versus 0.53, P<0.001) scores. Improvements in discrimination were also observed for the other outcomes.Conclusions:We derived a model that utilizes clinical text from paramedic reports to identify stroke. Our results require validation but have the potential of improving prehospital routing protocols.

Experiments are often used to produce emulators of deterministic computer code. This article introduces composite grid experimental designs and a sequential method for building the designs for accurate emulation. Computational methods are developed that enable fast and exact Gaussian process inference even with large sample sizes. We demonstrate that the proposed approach can produce emulators that are orders of magnitude more accurate than current approximations at a comparable computational cost.

Introduction: Acute stroke (AS) is a highly time sensitive treatment condition affecting approximately 800,000 people/year in the US. Most AS patients receive care at a primary stroke center (PSC), but some require more advanced treatments, and rely on a timely transfer to a comprehensive stroke center (CSC) where such treatments can be given. Stroke teams at 2 Chicago area PSCs and 4 CSCs, collectively, developed solutions (Graph) targeting both reported and perceived failures/delays/weakness in the current PSC door-in-door-out (DIDO) process for transferring patients to a CSC. The study simulates the potential impact of the solutions on DIDO. Methods: Current state (baseline) times were calculated from time stamps in the electronic health record (EHR) (e.g., door to CT), estimated by the stroke teams (e.g., hand-off time) or retrieved (e.g., DIDO, door to stroke activation) from a prospectively maintained REDCap data registry (2/2018-1/2020). Proportions (e.g., % with ischemic stroke, % transferred) were estimated from hospital data. Changes in times after implementation of a solution were obtained from peer reviewed literature, when available, or by consensus expert opinion. Simio (version 11.197.19514) was used to simulate the current and future states with implementation of the solutions, with 500 replications, to estimate changes in DIDO. Results: Implementation of all solutions would achieve a decrease in DIDO of 33 minutes (19%) from current state. The largest driver of this change was direct to CT/CTA protocol implementation (21 minutes) followed by using a handoff tool for paramedics prior to transfer (13 minutes). Conclusion: The proposed solutions can achieve nearly a 20% reduction in DIDO times. The “Direct to CT/CTA Protocol” solution is the major driver of the improvement. Data simulation is helpful by assessing the potential impact of many solutions and the relative impact of each solution to inform implementation decisions.

Computer simulation experiments are commonly used as an inexpensive alternative to real-world experiments to form a metamodel that approximates the input-output relationship of the real-world experiment. While a user may want to understand the entire response surface, they may also want to focus on interesting regions of the design space, such as where the gradient is large. In this paper we present an algorithm that adaptively runs a simulation experiment that focuses on finding areas of the response surface with a large gradient while also gathering an understanding of the entire surface. We consider the scenario where small batches of points can be run simultaneously, such as with multi-core processors.

Introduction: Early identification of stroke by emergency medical services (EMS) providers in the prehospital setting is associated with increased treatment rates, improved functional outcomes, and reduced mortality. We hypothesize that a predictive model utilizing machine learning and natural language processing (NLP) techniques can be developed to analyze EMS run reports to identify stroke patients accurately. Methods: We analyzed EMS data from the Chicago Fire Department matched with inpatient data on confirmed and suspected strokes from 17 Chicago hospitals in the Get With The Guidelines-Stroke (GWTG-Stroke) registry from 11/28/2018 to 5/31/2019. Using features derived from paramedic notes, we developed a support vector machine classifier to predict the following categories: any stroke, AIS-LVO, severe stroke (NIHSS>5), and CSC-eligible stroke (AIS-LVO or ICH/SAH). Individuals were randomly assigned into model derivation (70%) and validation cohorts (30%). C-statistics were used to evaluate discrimination of the classifier for stroke categories. Results: A total of 965 patients were included for analysis. In a validation cohort of 289 patients, the text-based model predicted stroke better than models trained using the Cincinnati Prehospital Stroke Scale (CPSS, c-statistic: 0.73 vs. 0.67, P=0.165) and the 3-Item Stroke Scale (3I-SS, c-statistic: 0.73 vs. 0.53, P <0.001) scores. The text-based model also demonstrated improved performance over the CPSS and 3I-SS models in discriminating patients with other stroke categories (Table 1). Conclusion: We derived a predictive model using clinical text from paramedic reports that has superior performance to existing prehospital clinical screening tools to identify stroke in the prehospital setting. Future studies can evaluate the implementation of an NLP-based decision tool to assist in prehospital stroke evaluation and destination decision-making.

Introduction: Some acute stroke (AS)patients require transfer to comprehensive stroke centers (CSCs) for time-sensitive, advanced treatments that lead to better outcomes. However, door-in-door-out (DIDO). PSC DIDO processes at primary stroke centers (PSCs) can be prolonged and result in delay or failure to deliver the advanced treatments at the CSC. We simulated the impact of reducing PSC DIDO times on the rate of inappropriate transfers to CSCs, a potential consequence of such efforts. Methods: Clinicians from3Chicago-area CSCs and 3 affiliated PSCs and the two main ambulance providers created a PSC DIDO process map Patient-level data from the 3 PSCs (N-108) and estimates from the literature were used to determine the distribution, range, or proportion of each step in the process. Datainputs were varied using Python™ in simulations with 100 replications. Outputswere DIDO time, % patients transferred to CSC, and % inappropriate CSC transfers. Sensitivity analyses assessed most impactful factors on DIDO time. Results: Three key decision points for transfer of an AS patient to a CSC were identified: (1) After stroke code activation; (2) After telestroke consultation; and (3) Post-tPA administration(most common current process). The figureshows that increasing PSC transfer ratesimmediately after stroke code activation by 5% decreased DIDO time by 10 minutes (11%), while increasing the inappropriate CSC transfer rate by 4%. Sensitivity analyses show that total DIDO time is most sensitive to the proportion of PSC hemorrhagic strokes, accuracy of AS detection at triage, and proportion of AS patients arriving by EMS. Conclusions: A strategy of earlier detection of acute stroke at triage, followed rapidly by stroke code activation and streamlined transfer of AS patients is likely to benefit from CSC care is predicted to a large impact on DIDO times, with a very small trade-off in increasing inappropriate transfers.

Diabetes mellitus (DM) self-care teaching is often provided at point of care, but “clinician-centered” feedback about teaching is lacking. Semi-structured, facilitated interviews of front-line clinicians involved in DM care (e.g., meter, medications, injection technique) were conducted. Clinicians were asked their perspectives on the needs of newly diagnosed DM patients, specifically during transitions of care (e.g., discharge home). Clinicians were also asked to review current DM education materials (training supplies, print handouts) and subsequently developed potential solutions. Interview transcripts were independently coded by 3 coders, using a constant comparative method, to identify themes. MAXQDA software was used. Key themes were additionally audited by a diabetologist. Eleven sessions were conducted with clinicians (N=14, 2 certified DM educators, 4 endocrinology MD fellows, 3 advanced practice providers, 1 DM nurse, 2 pharmacists, 2 internal medicine MDs). The most commonly identified theme across clinician type was simplified, understandable content (8/11 sessions). Participants identified lack of centralization of supplies and education materials as the most significant barrier (9/11 sessions) to optimal care. Other top themes were insurance coverage for diabetes supplies and medications, need for customization, and limited access to diabetes educators. The most preferred solution was use of patient simulation (repetitive practice) [7/11 sessions], followed by teaching videos (5/11 sessions). Clinicians preferred comprehensive but simplified, understandable, customizable, diabetes teaching content. The lack of centralization of supplies and content and insurance coverage for supplies/medications are major barriers for clinicians to deliver optimal DM self-care teaching. These data also suggest that use of simulation could facilitate improvement in DM teaching by front-line clinicians. Disclosure K. Coyne: None. S. Hakimian: None. T. Pollack: None. S. Karam: None. G. Prince: None. E.K. Touma: None. D.W. Gatchell: None. R. Khorzad: None. B. Ankenman: None. J.L. Holl: None. A. Wallia: Research Support; Self; Eli Lilly and Company, Novo Nordisk Inc., UnitedHealth Group. Funding American Diabetes Association (1-13-JF-54 to A.W.); Chicago Center for Diabetes Translation Research/National Institute of Diabetes and Digestive and Kidney Diseases (P30DK092949); Agency for Healthcare Research and Quality (5R18HS026143-02)

NOTE: The first page of text has been automatically extracted and included below in lieu of an abstract second quarter. However, the automotive projects themselves were not well supported by clients. The intention was to have clients who were employees at a large (but distant) automotive interior design firm. However, these employees simply did not have enough time to work with the 100 teams of students and EDC faculty ended up acting as clients for most of these projects. Students were asked to find their own users which worked well for some projects and poorly for others. Another issue with both the websites and the automotive projects was that they were primarily commercially based, leading some students and faculty to desire projects that were more socially conscious. The problem for the EDC faculty was now fairly well posed, “For the first quarter of EDC, we want a large group of projects that have similar theme. The projects will allow students to have external, knowledgeable clients and access to a pool of potential users. Finally, the projects will be physical in nature, small-scale, able to be built by freshman in 10 weeks, and preferably have a social benefit.” Introduction to RIC and RRTC The Rehabilitation Institute of Chicago (RIC) is the leading rehabilitation hospital in the US (as ranked by US News and World Report for the last 15 years). They have a plethora of programs for treatment of complex conditions including cerebral palsy, spinal cord injury, stroke and traumatic brain injury, as well as more common conditions such as arthritis, chronic pain and sports injuries. RIC also provides specialized services such as Assistive Technology, Prosthetics and Orthotics and Vocational Rehabilitation which help individuals of all ages lead more independent and fulfilling lives. In 2002, RIC and Northwestern received funding from NIDRR to operate research and training projects under the Rehabilitation and Research Training Center (RRTC). The center is focused on finding the technology-assisted solutions for stroke survivors to achieve independence in their home and community environments. Part of the mission of this center is to engage engineering students in the development of novel designs to assist in accomplishing common tasks that prove difficult for stroke survivors. Patients with disabilities present a variety of physical and emotional needs, many of which can be readily addressed by engineers. Stroke causes a number of specific problems. Weakness or paralysis of the arm and leg are the most comment effects, but other problems including sensory changes, speech and language disorders, cognitive deficits, swallowing dysfunction, and visual changes, also affect a stroke patient's ability to function. These deficits cause disabilities in daily functioning such as dressing, bathing, and walking. Rehabilitation consists of measures to improve the patient's ability to perform activities of daily living, including dressing, bathing, and walking, through training, supervised practice, counseling, and the use of specialized equipment. The center also is charged with increasing the awareness of the great potential for rehabilitation for stroke survivors. Student design projects can also aid in achieving this second goal since students get first hand knowledge and experience with the effects and rehabilitation of stroke survivors.

Introduction: Given the time-sensitive benefits of acute stroke (AS) treatments, stroke systems of care must balance reducing door-in-door-out (DIDO) time at primary stroke centers (PSCs) with capacity limits at comprehensive stroke centers (CSCs). For example transferring more AS patients earlier in the process (e.g., prior vascular imaging for large vessel occlusion) from PSCs would result in more inappropriate transfers to CSCs that could overburden these centers.We conducted a simulation to estimate the balance between increased AS transfers from PSCs to CSCs and the percent of CSC time on “bypass” (inability to accept transfers to neuro-ICU). Methods: Clinicians from 3 Chicago-area CSCs and 3 affiliated PSCs and the Chicago Emergency Medical Services (EMS) created a PSC DIDO process map. We assumed CSC time on bypass is affected by AS and non-AS admissions from the CSC and from the affiliated PSCs. Input data were obtained fromtheChicago region registry (e.g., # PSC to CSC transfers), peer reviewed literature (US average transfer rate of AS patients to CSCs), EMS (PSC-CSC affiliations), and CSCs (e.g., average bed occupancy rates). CSC size was estimated by #neuro-ICU beds: small (12 beds), medium (23 beds), and large (28 beds). The simulation output was % time of CSC on “bypass”. Results: Table shows % time of CSC on bypass by varying PSC AS transfer rates for each category of CSC size. Larger increases in PSC transfer rates resulted in modest increases in CSC bypass rates, particularly for medium and large CSCs. Validation with data from one CSC showed < 4% overestimate of CSC % time on bypass. Conclusion: CSCs with more beds have efficiencies of scale leading to lower % time on bypass, even with increases in PSC AS transfer rates proportionate to CSC size. This model allows stroke systems of care to compute regional CSCs’ % time on bypass based on actual PSCs’ transfer rates and CSC size.

This data article provides the summary data from tests comparing various Gaussian process software packages. Each spreadsheet represents a single function or type of function using a particular input sample size. In each spreadsheet, a row gives the results for a particular replication using a single package. Within each spreadsheet there are the results from eight Gaussian process model-fitting packages on five replicates of the surface. There is also one spreadsheet comparing the results from two packages performing stochastic kriging. These data enable comparisons between the packages to determine which package will give users the best results.

Gaussian process fitting, or kriging, is often used to create a model from a set of data. Many available software packages do this, but we show that very different results can be obtained from different packages even when using the same data and model. Seven different fitting packages that run on four different platforms are compared using various data functions and data sets that reveal there are stark differences between the packages. In addition to comparing the prediction accuracy, the predictive variance---which is important for evaluating precision of predictions and is often used in stopping criteria---is also evaluated.

As the service industry moves toward self-service, peer feedback serves a critical role in this shift for educational services. Peer feedback is a process by which students provide feedback to each other. One of its major benefits is that it enables students to become actively involved in the learning and assessment process and play an integral role in the delivery and quality of their education. However, a primary concern is that students do not consistently provide each other with quality feedback, especially in science, technology, engineering, and mathematics (STEM) disciplines in which gender stereotypes may hinder the ability of women to provide critical peer feedback. A potential way to improve peer feedback is to create anonymous review settings. This study examines how anonymity alters the nature of peer feedback in a large introductory undergraduate statistics class for computer science and engineering majors. In this class, peers review a series of team video projects as either anonymous or nonanonymous reviewers. Our results show that female peer reviewers were more affected by the anonymity setting than the male peer reviewers. We discuss the implications of these findings for promoting greater participation and retention of women in underrepresented STEM disciplines and the design of effective peer-review processes for improved student achievement and satisfaction.

The mean of the output of interest obtained from a run of a computer simulation model of a system or process often depends on many factors; many times, however, only a few of these factors are important. Sequential bifurcation is a method that has been considered by several authors for identifying these important factors using as few runs of the simulation model as possible. In this article, we propose a new sequential bifurcation procedure whose steps use a key stopping rule that can be calculated explicitly, something not available in the best methods previously considered. Moreover, we show how this stopping rule can also be easily modified to efficiently identify those factors that are important in influencing the variability rather than the mean of the output. In empirical studies, the new method performs better than previously published fully sequential bifurcation methods in terms of achieving the prescribed Type I error. It also achieves higher power for detecting moderately large effects using fewer replications than earlier methods. To achieve this control for midrange effects, the new method sometimes requires more replications than other methods in the case where there are many very large effects.

When fitting complex models, such as finite element or discrete event simulations, the experiment design should exhibit desirable properties of both projectivity and orthogonality. To reduce experimental effort, sequential design strategies allow experimenters to collect data only until some measure of prediction precision is reached. In this article, we present a batch sequential experiment design method that uses sliced full factorial-based Latin hypercube designs (sFFLHDs), which are an extension to the concept of sliced orthogonal array-based Latin hypercube designs (OALHDs). At all stages of the sequential design, good univariate stratification is achieved. The structure of the FFLHDs also tends to produce uniformity in higher dimensions, especially at certain stages of the design. We show that our batch sequential design approach has good sampling and fitting qualities through both empirical studies and theoretical arguments. Supplementary materials are available online.