Iterative Design of a Clinical Decision Support System Informed by a Mixed Methodology in the Management of Geriatric Acute Abdominal Pain

Clinical decision support systems (CDSS) have the potential to impact provider behavior. Optimizing CDSS-provider interaction may enhance CDSS use. Five CDSS pain management interventions were developed and deployed in test scenarios within a simulated EHR that mirrored typical Emergency Department (ED) workflow. Provider feedback was analyzed using a mixed methodology approach. The CDSS interventions were iteratively designed across three rounds of testing based upon this analysis. Iterative design led to improved provider usability and favorability scores. Background: As the United States population ages, a greater proportion of ED visits will encompass geriatric patients. Examination of ED care of the geriatric population has revealed challenges in addressing pain symptoms. Unrelieved acute pain among geriatric patients is associated with poor outcomes such as increased morbidity and greater length of hospital stay. CDSS offer a promising approach to improve provider behavior in managing elder oligoanalgesia. Evaluation of CDSS within the normal provider workflow is often overlooked in the development of informatics interventions. This evaluation may help refine CDSS to reduce alert fatigue and frustration, thereby making CDSS a more valuable tool in guiding clinical care. This study sought to refine and optimize a geriatric abdominal pain care CDSS by using an iterative design process utilizing a test EHR environment. Methods: A group of five CDSS interventions was developed to address geriatric acute abdominal pain care throughout an ED visit (Table 1). Thirteen emergency physicians, all experienced in using the EHR, were recruited for participation. Over a 10-week period, seventeen 1-hour usability test sessions were conducted across 3 rounds of testing [Round 1: 5 users; Round 2: 5 users (1 repeat user from Round 1); Round 3: 7 users (2 repeat users from Round 2)]. Physicians were given 3 patient scenarios and were asked to provide simulated clinical care using the EHR, while interacting with the CDSS interventions. Users utilized order entry, documentation and discharge workflows. A System Usability Scale (SUS) survey (100 being perfect score), structured favorability questionnaire [scored negative (1), neutral (3), positive (5)], and open-ended narrative feedback of each CDSS intervention were completed after each user session. An interdisciplinary team reviewed structured and unstructured feedback and favorability scores after each round of testing, identified positive and negative issues in effectiveness, efficiency, and satisfaction, and then incorporated changes to the CDSS design. The SUS score was calculated for each round to quantify the overall effectiveness, efficiency and satisfaction for the CDSS. CDSS Intervention Description Type of Intervention Pain Score of 10 Alert A visual pop-up graphic in the center of the EHR screen that interrupts workflow and alerts provider that patient has pain score of 10 that has not been addressed Interruptive Alert Revaluate at 4 Hours Alert A visual stimulus within the order entry screen of the EHR that alerts provider that patient has pain score of 10 that has not been re-evaluated or addressed after 4 hours Non-Interruptive Alert Order Set A hyperlink that offers the provider a set of predefined analgesic treatment options for geriatric patients Non-Interruptive, interactive Medication Decision Aide HPI Reminder A statement that appears in history of present illness documentation reminding the provider to address patient’s pain Non-Interruptive, Non-Interactive Reminder or ‘Nudge’ Alert At Discharge A grayed out print button that prevents printing of discharge instructions for patients with an unaddressed pain score of 10 Interruptive Hard-Stop Alert Table 1. Description of Elderly Acute Abdominal Pain Care CDSS Interventions Figure 2. Example of CDSS iterative design over 3 rounds. There has been streamlining of text, clearer buttons and direct links to order sets or pain re-evaluation dialogs. (Four additional CDSS interventions not pictured due to space constraints on abstract) Results: After each round of CDSS evaluation, favorability scores and unstructured feedback responses were used to prioritize and target areas to improve usability, effectiveness, and the efficacy of the five CDSS interventions. See Figure 1 for a representative example of the changes that occurred (e.g., reduced text, fewer required fields, and direct links to actionable items) with the intervention for managing patient-reported pain scores of 10. Over the three testing rounds and redesigns, mean SUS scores improved as did mean favorability scores [scale 1 (worsened care) to 5 (improved care)] (Table 1). Round SUS Mean SUS Min SUS Max Fav. Mean Fav. Min Fav. Max n 1 75 64 87 3.4 2.2 4.2 5 2 81 70 96 2.9 1.8 3.8 5 3 89 69 100 4.5 4.4 4.7 7 Table 1. Mean System Usability Scores (SUS) and Favorability scores (Fav.) with minimum and maximum scores across three usability rounds showing iterative improvement. Limitations: Human-computer interactions are complex and may be considered in the larger socio-technical model. Although in this study EHR workflow was designed to mimic ED workflow, this study was conducted in an artificial environment that may not account for various other factors such as monitor alarms, patients and telephones competing for clinician’s attention. Though usability testing does not typically comprise a large number of trials, this study would be strengthened by a larger number of users completing the testing, none of who had participated in prior iterative rounds of testing. Though this study was conducted in one widely utilized commercial enterprise EHR, the interventions and effect may not transfer to other EHRs Conclusion: This study demonstrates how an iterative design process based on mixed methodology and implemented by an interdisciplinary team comprised of clinicians, research associates, and hospital information technology improves the usability and favorability of a geriatric pain care CDSS among providers. This process may be utilized in other institutions to improve provider satisfaction and enhance the use of CDSS. Clinical leadership has approved this CDSS implementation in the Mount Sinai Geriatric ED. Future studies will include real-world testing of the interventions to determine provider efficiency and clinical impact. Acknowledgements: This study was supported by: NIA grant # 5R21AG040734-02, Mount Sinai GCO# 10-1414(0001)(02)EM and New York State Empire Clinical Research Investigator Program