ObjectivesThe Computer-Aided Risk Score for Mortality (CARM) estimates the risk of in-hospital mortality following acute admission to the hospital by automatically amalgamating physiological measures, blood tests, gender, age and COVID-19 status. Our aims were to implement the score with a small group of practitioners and understand their first-hand experience of interacting with the score in situ.DesignPilot implementation evaluation study involving qualitative interviews.SettingThis study was conducted in one of the two National Health Service hospital trusts in the North of England in which the score was developed.ParticipantsMedical, older person and ICU/anaesthetic consultants and specialist grade registrars (n=116) and critical outreach nurses (n=7) were given access to CARM. Nine interviews were conducted in total, with eight doctors and one critical care outreach nurse.InterventionsParticipants were given access to the CARM score, visible after login to the patients' electronic record, along with information about the development and intended use of the score.ResultsFour themes and 14 subthemes emerged from reflexive thematic analysis: (1) current use (including support or challenge clinical judgement and decision making, communicating risk of mortality and professional curiosity); (2) barriers and facilitators to use (including litigation, resource needs, perception of the evidence base, strengths and limitations), (3) implementation support needs (including roll-out and integration, access, training and education); and (4) recommendations for development (including presentation and functionality and potential additional data). Barriers and facilitators to use, and recommendations for development featured highly across most interviews.ConclusionOur in situ evaluation of the pilot implementation of CARM demonstrated its scope in supporting clinical decision making and communicating risk of mortality between clinical colleagues and with service users. It suggested to us barriers to implementation of the score. Our findings may support those seeking to develop, implement or improve the adoption of risk scores.

Background: Patients are increasingly being asked to provide feedback about their experience of health-care services. Within the NHS, a significant level of resource is now allocated to the collection of this feedback. However, it is not well understood whether or not, or how, health-care staff are able to use these data to make improvements to future care delivery. Objective: To understand and enhance how hospital staff learn from and act on patient experience (PE) feedback in order to co-design, test, refine and evaluate a Patient Experience Toolkit (PET). Design: A predominantly qualitative study with four interlinking work packages. Setting: Three NHS trusts in the north of England, focusing on six ward-based clinical teams (two at each trust). Methods: A scoping review and qualitative exploratory study were conducted between November 2015 and August 2016. The findings of this work fed into a participatory co-design process with ward staff and patient representatives, which led to the production of the PET. This was primarily based on activities undertaken in three workshops (over the winter of 2016/17). Then, the facilitated use of the PET took place across the six wards over a 12-month period (February 2017 to February 2018). This involved testing and refinement through an action research (AR) methodology. A large, mixed-methods, independent process evaluation was conducted over the same 12-month period. Findings: The testing and refinement of the PET during the AR phase, with the mixed-methods evaluation running alongside it, produced noteworthy findings. The idea that current PE data can be effectively triangulated for the purpose of improvement is largely a fallacy. Rather, additional but more relational feedback had to be collected by patient representatives, an unanticipated element of the study, to provide health-care staff with data that they could work with more easily. Multidisciplinary involvement in PE initiatives is difficult to establish unless teams already work in this way. Regardless, there is merit in involving different levels of the nursing hierarchy. Consideration of patient feedback by health-care staff can be an emotive process that may be difficult initially and that needs dedicated time and sensitive management. The six ward teams engaged variably with the AR process over a 12-month period. Some teams implemented far-reaching plans, whereas other teams focused on time-minimising ‘quick wins’. The evaluation found that facilitation of the toolkit was central to its implementation. The most important factors here were the development of relationships between people and the facilitator’s ability to navigate organisational complexity. Limitations: The settings in which the PET was tested were extremely diverse, so the influence of variable context limits hard conclusions about its success. Conclusions: The current manner in which PE feedback is collected and used is generally not fit for the purpose of enabling health-care staff to make meaningful local improvements. The PET was co-designed with health-care staff and patient representatives but it requires skilled facilitation to achieve successful outcomes. Funding: The National Institute for Health Research Health Services and Delivery Research programme.

Objectives The Computer-Aided Risk Score (CARS) estimates the risk of death following emergency admission to medical wards using routinely collected vital signs and blood test data. Our aim was to elicit the views of healthcare practitioners (staff) and service users and carers (SU/C) on (1) the potential value, unintended consequences and concerns associated with CARS and practitioner views on (2) the issues to consider before embedding CARS into routine practice. Setting This study was conducted in two National Health Service (NHS) hospital trusts in the North of England. Both had in-house information technology (IT) development teams, mature IT infrastructure with electronic National Early Warning Score (NEWS) and were capable of integrating NEWS with blood test results. The study focused on emergency medical and elderly admissions units. There were 60 and 39 acute medical/elderly admissions beds at the two NHS hospital trusts. Participants We conducted eight focus groups with 45 healthcare practitioners and two with 11 SU/Cs in two NHS acute hospitals. Results Staff and SU/Cs recognised the potential of CARS but were clear that the score should not replace or undermine clinical judgments. Staff recognised that CARS could enhance clinical decision-making/judgments and aid communication with patients. They wanted to understand the components of CARS and be reassured about its accuracy but were concerned about the impact on intensive care and blood tests. Conclusion Risk scores are widely used in healthcare, but their development and implementation do not usually involve input from practitioners and SU/Cs. We contributed to the development of CARS by eliciting views of staff and SU/Cs who provided important, often complex, insights to support the development and implementation of CARS to ensure successful implementation in routine clinical practice.

OBJECTIVES:There is growing evidence that patients can provide feedback on the safety of their care. The 44-item Patient Measure of Safety (PMOS) was developed for this purpose. While valid and reliable, the length of this questionnaire makes it potentially challenging for routine use. Our study aimed to produce revised, shortened versions of PMOS (PMOS-30 and PMOS-10), which retained the psychometric properties of the longer version.PARTICIPANTS:To produce a shortened diagnostic measure, we analysed data from 2002 patients who completed PMOS-44, and examined the reliability of the revised measure (PMOS-30) in a sample of 751 patients. To produce a brief standalone measure, we again analysed data from 2002 patients who completed PMOS-44, and tested the reliability and validity of the brief standalone measure (PMOS-10) in a sample of 165 patients.METHODS:The process of shortening the questionnaire involved a combination of secondary data analysis (eg, Standard Deviation and inter-item correlations) and a consensus group exercise to produce PMOS-30 and examine face validity. Analysis of PMOS-30 data examined reliability (eg, Cronbach's alpha). Further secondary data analysis (ie, corrected item-total correlations) produced PMOS-10, and primary data collection assessed its reliability and validity (eg, Cronbach's alpha, analysis of variance).RESULTS:Fourteen items were removed to produce PMOS-30 and the percentage of negatively worded items was reduced from 57% to 33%. PMOS-30 demonstrated good internal reliability (α=0.89). The 10 items with the highest corrected item-total correlations across both PMOS-44 and PMOS-30 composed PMOS-10. PMOS-10 had good internal reliability (α=0.79), demonstrated convergent validity; however, discriminant validity was not established.CONCLUSIONS:Two revised, shortened versions of the original PMOS-44 (PMOS-30 and PMOS-10) were produced to capture patient feedback about safety in hospital. The measures demonstrated good reliability and validity, and preserved the psychometric properties of the original measure.

Longitudinal qualitative research is starting to be used in applied health research, having been popular in social research for several decades. There is potential for a large volume of complex data to be captured, over a span of months or years across several different methods. How to analyse this volume of data – with its inherent complexity - represents a problem for health researchers. There is a previous dearth of methodological literature which describes an appropriate analytic process which can be readily employed. We document a worked example of the Pen Portrait analytic process, using the qualitative dataset for which the process was originally developed. Pen Portraits are recommended as a way in which longitudinal health research data can be concentrated into a focused account. The four stages of undertaking a pen portrait are: 1) understand and define what to focus on 2) design a basic structure 3) populate the content 4) interpretation. Instructive commentary and guidance is given throughout with consistent reference to the original study for which Pen Portraits were devised. The Pen Portrait analytic process was developed by the authors, borne out of a need to effectively integrate multiple qualitative methods collected over time. Pen Portraits are intended to be adaptable and flexible, in order to meet the differing analytic needs of qualitative longitudinal health studies. The Pen Portrait analytic process provides a useful framework to enable researchers to conduct a robust analysis of multiple sources of qualitative data collected over time.

Context Collecting feedback from patients about their experiences of health care is an important activity. However, improvement based on this feedback rarely materializes. In this study, we focus on answering the question-"what is impeding the use of patient experience feedback?" Methods We conducted a qualitative study in 2016 across three NHS hospital Trusts in the North of England. Focus groups were undertaken with ward-based staff, and hospital managers were interviewed in-depth (50 participants). We conducted a conceptual-level analysis. Findings On a macro level, we found that the intense focus on the collection of patient experience feedback has developed into its own self-perpetuating industry with a significant allocation of resource, effort and time being expended on this task. This is often at the expense of pan-organizational learning or improvements being made. On a micro level, ward staff struggled to interact with feedback due to its complexity with questions raised about the value, validity and timeliness of data sources. Conclusions Macro and micro prohibiting factors come together in a perfect storm which provides a substantial impediment to improvements being made. Recommendations for policy change are put forward alongside recognition that high-level organizational culture/systems are currently too sluggish to allow fruitful learning and action to occur from the feedback that patients give.

AbstractBackground & objectivesThe comparative uses of different types of patient experience (PE) feedback as data within quality improvement (QI) are poorly understood. This paper reviews what types are currently available and categorizes them by their characteristics in order to better understand their roles in QI.MethodsA scoping review of types of feedback currently available to hospital staff in the UK was undertaken. This comprised academic database searches for “measures of PE outcomes” (2000‐2016), and grey literature and websites for all types of “PE feedback” potentially available (2005‐2016). Through an iterative consensus process, we developed a list of characteristics and used this to present categories of similar types.Main resultsThe scoping review returned 37 feedback types. A list of 12 characteristics was developed and applied, enabling identification of 4 categories that help understand potential use within QI—(1) Hospital‐initiated (validated) quantitative surveys: for example the NHS Adult Inpatient Survey; (2) Patient‐initiated qualitative feedback: for example complaints or twitter comments; (3) Hospital‐initiated qualitative feedback: for example Experience Based Co‐Design; (4) Other: for example Friends & Family Test. Of those routinely collected, few elicit “ready‐to‐use” data and those that do elicit data most suitable for measuring accountability, not for informing ward‐based improvement. Guidance does exist for linking collection of feedback to QI for some feedback types in Category 3 but these types are not routinely used.ConclusionIf feedback is to be used more frequently within QI, more attention must be paid to obtaining and making available the most appropriate types.

BackgroundThe NHS Long Term Plan aims to make care more Patient-Centred through listening to patients. The acute nature of ED care presents barriers to collecting patient feedback. We have explored two interventions (PRASE and Y-PET), as mechanisms for collecting and reporting safety feedback (PRASE) and experience feedback (Y-PET) in EDs.An iterative approach was used to develop PMOS10 (PRASE questionnaire) for the ED which was tested in 5 departments with over 100 patients. The Y-PET was used alongside these tools in 2 departments with 40 patients.A mixture of patient volunteers and staff collected the feedback.Two questions from the PMOS10 proved to be unsuitable for the ED setting, and were substituted. Through further iterative tests, we now have a PMOS10(ED).Hospital volunteers and staff not associated with the department are best placed to collect unbiased results. Patients (or their relatives) who are awaiting transport home or a hospital bed are best placed to give feedback. The traffic light display of patient safety feedback provided in PRASE is useful for assurance but staff need more qualitative data to inspire change. PRASE patient comments go someway to providing this but are strongly enhanced by the open answers of the Y-PET. The Y-PET format for presenting qualitative data as headline areas to celebrate or improve was effective in engaging staff in feedback from both tools.ED patients can give valuable insight into how safe their care is and areas to celebrate and improve. Staff can engage with feedback themes and key quotes to initiate improvement.

OBJECTIVES:There are no established mortality risk equations specifically for emergency medical patients who are admitted to a general hospital ward. Such risk equations may be useful in supporting the clinical decision-making process. We aim to develop and externally validate a computer-aided risk of mortality (CARM) score by combining the first electronically recorded vital signs and blood test results for emergency medical admissions.DESIGN:Logistic regression model development and external validation study.SETTING:Two acute hospitals (Northern Lincolnshire and Goole NHS Foundation Trust Hospital (NH)-model development data; York Hospital (YH)-external validation data).PARTICIPANTS:Adult (aged ≥16 years) medical admissions discharged over a 24-month period with electronic National Early Warning Score(s) and blood test results recorded on admission.RESULTS:The risk of in-hospital mortality following emergency medical admission was 5.7% (NH: 1766/30 996) and 6.5% (YH: 1703/26 247). The C-statistic for the CARM score in NH was 0.87 (95% CI 0.86 to 0.88) and was similar in an external hospital setting YH (0.86, 95% CI 0.85 to 0.87) and the calibration slope included 1 (0.97, 95% CI 0.94 to 1.00).CONCLUSIONS:We have developed a novel, externally validated CARM score with good performance characteristics for estimating the risk of in-hospital mortality following an emergency medical admission using the patient's first, electronically recorded, vital signs and blood test results. Since the CARM score places no additional data collection burden on clinicians and is readily automated, it may now be carefully introduced and evaluated in hospitals with sufficient informatics infrastructure.

BACKGROUND:Patient safety measurement remains a global challenge. Patients are an important but neglected source of learning; however, little is known about what patients can add to our understanding of safety. We sought to understand the incidence and nature of patient-reported safety concerns in hospital.METHODS:Feedback about the experience of safety within hospital was gathered from 2471 inpatients as part of a multicentre, waitlist cluster randomised controlled trial of an intervention, undertaken within 33 wards across three English NHS Trusts, between May 2013 and September 2014. Patient volunteers, supported by researchers, developed a classification framework of patient-reported safety concerns from a random sample of 231 reports. All reports were then classified using the patient-developed categories. Following this, all patient-reported safety concerns underwent a two-stage clinical review process for identification of patient safety incidents.RESULTS:Of the 2471 inpatients recruited, 579 provided 1155 patient-reported incident reports. 14 categories were developed for classification of reports, with communication the most frequently occurring (22%), followed by staffing issues (13%) and problems with the care environment (12%). 406 of the total 1155 patient incident reports (35%) were classified by clinicians as a patient safety incident according to the standard definition. 1 in 10 patients (264 patients) identified a patient safety incident, with medication errors the most frequently reported incident.CONCLUSIONS:Our findings suggest that patients can provide insight about safety that complements existing patient safety measurement, with a frequency of reported patient safety incidents that is similar to those obtained via case note review. However, patients provide a unique perspective about hospital safety which differs from and adds to current definitions of patient safety incidents.TRIAL REGISTRATION NUMBER:ISRCTN07689702; pre-results.

Patients are increasingly being asked for feedback about their healthcare experiences. However, healthcare staff often find it difficult to act on this feedback in order to make improvements to services. This paper draws upon notions of legitimacy and readiness to develop a conceptual framework (Patient Feedback Response Framework PFRF) which outlines why staff may find it problematic to respond to patient feedback.A large qualitative study was conducted with 17 ward based teams between 2013 and 2014, across three hospital Trusts in the North of England. This was a process evaluation of a wider study where ward staff were encouraged to make action plans based on patient feedback. We focus on three methods here: i) examination of taped discussion between ward staff during action planning meetings ii) facilitators notes of these meetings iii) telephone interviews with staff focusing on whether action plans had been achieved six months later. Analysis employed an abductive approach.Through the development of the PFRF, we found that making changes based on patient feedback is a complex multi-tiered process and not something that ward staff can simply 'do'. First, staff must exhibit normative legitimacy the belief that listening to patients is a worthwhile exercise. Second, structural legitimacy has to be in place ward teams need adequate autonomy, ownership and resource to enact change. Some ward teams are able to make improvements within their immediate control and environment. Third, for those staff who require interdepartmental co-operation or high level assistance to achieve change, organisational readiness must exist at the level of the hospital otherwise improvement will rarely be enacted. Case studies drawn from our empirical data demonstrate the above. It is only when appropriate levels of individual and organisational capacity to change exist, that patient feedback is likely to be acted upon to improve services. (C) 2017 The Authors. Published by Elsevier Ltd.This is an open access article under the CC BY-NC-ND license.

Objectives A patient safety intervention was tested in a 33-ward randomised controlled trial. No statistically significant difference between intervention and control wards was found. We conducted a process evaluation of the trial and our aim in this paper is to understand staff engagement across the 17 intervention wards.Design Large qualitative process evaluation of the implementation of a patient safety intervention.Setting and participants National Health Service staff based on 17 acute hospital wards located at five hospital sites in the North of England.Data We concentrate on three sources here: (1) analysis of taped discussion between ward staff during action planning meetings; (2) facilitators' field notes and (3) follow-up telephone interviews with staff focusing on whether action plans had been achieved. The analysis involved the use of pen portraits and adaptive theory.Findings First, there were palpable differences in the ways that the 17 ward teams engaged with the key components of the intervention. Five main engagement typologies were evident across the life course of the study: consistent, partial, increasing, decreasing and disengaged. Second, the intensity of support for the intervention at the level of the organisation does not predict the strength of engagement at the level of the individual ward team. Third, the standardisation of facilitative processes provided by the research team does not ensure that implementation standardisation of the intervention occurs by ward staff.Conclusions A dilution of the intervention occurred during the trial because wards engaged with Patient Reporting and Action for a Safe Environment (PRASE) in divergent ways, despite the standardisation of key components. Facilitative processes were not sufficiently adequate to enable intervention wards to successfully engage with PRASE components.

Background: Estimates suggest that, in NHS hospitals, incidents causing harm to patients occur in 10% of admissions, with costs to the NHS of > £2B. About one-third of harmful events are believed to be preventable. Strategies to reduce patient safety incidents (PSIs) have mostly focused on changing systems of care and professional behaviour, with the role that patients can play in enhancing the safety of care being relatively unexplored. However, although the role and effectiveness of patient involvement in safety initiatives is unclear, previous work has identified a general willingness among patients to contribute to initiatives to improve health-care safety. Aim: Our aim in this programme was to design, develop and evaluate four innovative approaches to engage patients in preventing PSIs: assessing risk, reporting incidents, direct engagement in preventing harm and education and training. Methods and results: We developed tools to report PSIs [patient incident reporting tool (PIRT)] and provide feedback on factors that might contribute to PSIs in the future [Patient Measure of Safety (PMOS)]. These were combined into a single instrument and evaluated in the Patient Reporting and Action for a Safe Environment (PRASE) intervention using a randomised design. Although take-up of the intervention by, and retention of, participating hospital wards was 100% and patient participation was high at 86%, compliance with the intervention, particularly the implementation of action plans, was poor. We found no significant effect of the intervention on outcomes at 6 or 12 months. The ThinkSAFE project involved the development and evaluation of an intervention to support patients to directly engage with health-care staff to enhance their safety through strategies such as checking their care and speaking up to staff if they had any concerns. The piloting of ThinkSAFE showed that the approach is feasible and acceptable to users and may have the potential to improve patient safety. We also developed a patient safety training programme for junior doctors based on patients who had experienced PSIs recounting their own stories. This approach was compared with traditional methods of patient safety teaching in a randomised controlled trial. The study showed that delivering patient safety training based on patient narratives is feasible and had an effect on emotional engagement and learning about communication. However, there was no effect on changing general attitudes to safety compared with the control. Conclusion: This research programme has developed a number of novel interventions to engage patients in preventing PSIs and protecting them against unintended harm. In our evaluations of these interventions we have been unable to demonstrate any improvement in patient safety although this conclusion comes with a number of caveats, mainly about the difficulty of measuring patient safety outcomes. Reflecting this difficulty, one of our recommendations for future research is to develop reliable and valid measures to help efficiently evaluate safety improvement interventions. The programme found patients to be willing to codesign, coproduce and participate in initiatives to prevent PSIs and the approaches used were feasible and acceptable. These factors together with recent calls to strengthen the patient voice in health care could suggest that the tools and interventions from this programme would benefit from further development and evaluation. Trial registration: Current Controlled Trials ISRCTN07689702. Funding: The National Institute for Health Research Programme Grants for Applied Research programme.