Reliability of Evaluating Hospital Quality by Colorectal Surgical Site Infection Type

Objective: To determine whether risk-adjusted colorectal SSI rates are statistically reliable as hospital quality measures. Background: Policymakers use surgical site infections (SSI) for public reporting of hospital quality and pay-for-performance because they are a relatively common and costly cause of patient morbidity. Methods: Patients who underwent a colorectal procedure in 2009 were identified from the American College of Surgeons National Surgical Quality Improvement Program. We developed hierarchical multivariate logistic models for (1) superficial SSI, (2) deep/organ-space SSI, and (3) "any SSI" and compared how each model ranked hospital-level risk-adjusted performance. Statistical reliability of hospital quality measurements was estimated on a scale from 0 to 1; with 0 indicating that apparent variation between a hospital's quality measurement and the average hospital is statistically unreliable, and 1 indicating that any observed variation is due to a real difference in performance. Results: Mean reliability of hospital-level quality measurements was 0.650 for superficial, 0.404 for deep/organ-space, and 0.586 for "any SSI." Lower reliability was accounted for by relatively little variation in risk-adjusted SSI rates between hospitals and insufficient numbers of colorectal cases submitted by individual hospitals. In 2009, we estimate that 22.1% of all US hospitals performed a sufficient number of colorectal cases to report superficial SSI rates at a high standard of statistical reliability and 1.0% did for deep/organ-space SSI. Conclusions: As currently constructed, colorectal SSI quality measures might not meet a high standard of statistical reliability for most hospitals, limiting their ability to confidently differentiate high and low performance. Despite an expectation of improving statistical power, combining superficial and deep/organ-space SSI into an "any SSI" measure worsens reliability.