Reliability of Evaluating Hospital Quality Using Surgical Site Infections After Colorectal Procedures

Introduction: Policymakers are increasingly focusing on surgical site infections (SSI) for public reporting and pay-for-performance because they are a common and costly cause of patient morbidity. Our objective was to determine if superficial and deep/organ-space SSI occurring after colorectal procedures should be considered independently or aggregated when measuring hospital quality. Methods: Colorectal cases in the National Surgical Quality Improvement Program (NSQIP) were identified by CPT code, producing 25,455 cases from 236 hospitals in 2009. Hierarchical multivariable logistic models were developed for (1) superficial SSI, (2) deep/organ-space SSI and (3) all SSI. Models were adjusted for procedure and case-mix. for each model, hospitals' observed/expected (O/E) ratios were calculated then ranked and split into equal decile groups. Hospital decile was compared between the models. The reliability of each O/E was calculated on a scale from 0 to 1, with 0 indicating that variation is due to measurement error (“noise”) and 1 indicating that variation is due to real differences in performance (“truth”). The relationship between reliability and sample size was estimated using the Spearman-Brown prophecy. The 2009 Nationwide Inpatient Sample was used to estimate the number of colorectal procedures performed in US hospitals. Results: There was poor agreement between hospitals' decile rank using the superficial vs. deep/organ-space SSI models (weighted kappa 0.080). The mean O/E reliability was 0.731 for superficial, 0.620 for deep/organ-space, and 0.704 for all SSI. Reliability increased with increasing sample size (see figure). the estimated number of cases a hospital is projected to require for different levels of reliability and number of hospitals nationwide that performed the specified caseload in 2009 are reported in the table. Conclusions: for high-stakes quality comparisons such as pay-for-performance and public reporting it is essential that measures be valid and reliable - meaning high and low quality is accurately and consistently identified. We found a decrease in value when combining superficial and deep/organ-space SSI into one quality measure. Hospital risk-adjusted performance for these SSI types is not correlated, and grouping does not confer an advantage for the reliability of the quality measurement. Policies and initiatives aimed at reducing SSI should consider superficial and deep/organ-space SSI as two distinct entities when measuring quality. TableEstimated Number of Cases (n) a Hospital is Projected to Require for Different Levels of Reliability and Number of Hospitals Nationwide that Performed Specified Caseloads in 2009 Superficial SSI Model Deep/Organ-space SSI Model Any SSI Model Estimated number of cases (n) required for a given reliability: Rel=0.3 n=23 n=85 n=29 Rel=0.5 n=52 n=197 n=67 Rel=0.7 n=121 n=458 n=156 Rel=0.9 n=466 n=1,767 n=600 Estimated reliability for a given number of cases (n): # of US hospitals performing specified caseload n=20 0.279 0.092 0.231 2606 n=80 0.607 0.290 0.546 1251 n=100 0.659 0.338 0.600 1031 n=300 0.853 0.605 0.818 171 Total # of US hospitals that performed ≥1 colorectal case in 2009 = 3,970 Open table in a new tab Total # of US hospitals that performed ≥1 colorectal case in 2009 = 3,970