Development, implementation and evaluation of an early warning system improvement programme for children in hospital: the PUMA mixed-methods study

Background The Paediatric early warning system Utilisation and Morbidity Avoidance (PUMA) study was commissioned to develop, implement and evaluate a paediatric track-and-trigger tool for widespread adoption. Following findings from three systematic reviews, revised aims focused on implementation of a whole-systems improvement programme. Objectives (1) Identify, through systematic review, the following: evidence for core components of effective paediatric track-and-trigger tools and paediatric early warning systems, and contextual factors consequential for paediatric track-and-trigger tool and early warning system effectiveness. (2) Develop and implement an evidence-based paediatric early warning system improvement programme (i.e. the PUMA programme). (3) Evaluate the effectiveness of the PUMA programme by examining clinical practice and core outcomes trends. (4) Identify ingredients of successful implementation of the PUMA programme. Review methods The quantitative reviews addressed the following two questions: how well validated are existing paediatric track-and-trigger tools and their component parts for predicting inpatient deterioration? How effective are paediatric early warning systems (with or without a tool) at reducing mortality and critical events? The qualitative review addressed the following question: what sociomaterial and contextual factors are associated with successful or unsuccessful paediatric early warning systems (with or without tools)? Design Interrupted time series and ethnographic case studies were used to evaluate the PUMA programme. Qualitative methods were deployed in a process evaluation. Setting The study was set in two district general and two tertiary children’s hospitals. Intervention The PUMA programme is a paediatric early warning system improvement programme designed to harness local expertise to implement contextually appropriate interventions. Main outcome measures The primary outcome was a composite metric, representing children who experienced one of the following in 1 month: mortality, cardiac arrest, respiratory arrest, unplanned admission to a paediatric intensive care unit or unplanned admission to a high-dependency unit. Paediatric early warning system changes were assessed through ethnographic ward case studies. Results The reviews showed limited effectiveness of paediatric track-and-trigger tools in isolation, and multiple failure points in paediatric early warning systems. All sites made paediatric early warning system changes; some of the clearer quantitative findings appeared to relate to qualitative observations. Systems changed in response to wider contextual factors. Limitations Low event rates made quantitative outcome measures challenging. Implementation was not a one-shot event, creating challenges for the interrupted time series in conceptualising ‘implementation’ and ‘post-intervention’ periods. Conclusions Detecting and acting on deterioration in the acute hospital setting requires a whole-systems approach. The PUMA programme offers a framework to support ongoing system-improvement work; the approach could be used more widely. Organisational-level system change can affect clinical outcomes positively. Alternative outcome measures are required for research and quality improvement. Future work The following further research is recommended: a consensus study to identify upstream indicators of paediatric early warning system performance; an evaluation of OUTCOME approach in other clinical areas; an evaluation of supernumerary nurse co-ordinator role; and an evaluation of mandated system improvement. Study registration This study is registered as PROSPERO CRD42015015326. Funding This project was funded by the National Institute for Health Research (NIHR) Health and Social Care Delivery Research programme and will be published in full in Health and Social Care Delivery Research; Vol. 10, No. 1. See the NIHR Journals Library website for further project information.