A Comprehensive Evaluation of Statistical Reliability in ACS NSQIP Profiling Models

Objective: To assess statistical reliability of hospital profiling models in ACS NSQIP (American College of Surgeons' National Surgical Quality Improvement Program). Background: The ACS NSQIP January 2013 Semiannual Report provided risk-adjusted hospital quality assessments for 137 models. Methods: Median reliability and percentage of hospitals achieving acceptable reliability were computed for each model. Average median reliability was computed across models with common outcomes. Results: Median reliability varied across the 137 models, from a high of 0.91 for "All Cases Morbidity" to a low of 0.005 for "Procedure-Targeted Total Hip Arthroplasty Surgical Site Infection." Generally, reliability was greatest for models with larger sample sizes and higher outcome event rates. Among "Essentials" models, 72% attained a median reliability of 0.40 or more, and 24% of 0.70 or more. Among "Procedure-Targeted" models, 29% attained a median reliability of 0.40 or more, and 3% of 0.70 or more. Percentage of hospitals achieving an acceptable reliability of 0.40 ranged from 98% for "All Cases Morbidity" to 0% for "Procedure-Targeted Pancreatectomy Mortality." For Essentials models, average median reliability for each outcome, except mortality, was more than 0.40. However, for Procedure-Targeted models the average median was less than 0.40. Conclusions: For a large proportion of ACS NSQIP Essentials models, statistical reliability is adequate for assessing surgical quality and differentiating hospital performance. The Procedure-Targeted program is evolving in terms of statistical reliability, with promising results to date. These results also argue for broader discussions of statistical reliability in performance assessments for the profession.