The impact of different prognostic models and their customization on institutional comparison of intensive care units

Objectives. To evaluate the influence of choice of a prognostic model and the effect of customization of these models on league tables (i.e., rank-order listing) in which intensive care units (ICUs) are ranked by standardized mortality ratios using Acute Physiology and Chronic Health Evaluation (APACHE) 11, Simplified Acute Physiology Score (SAPS) 11, and Mortality Probability Model 11 (MPM,,11). Design. Retrospective analysis of prospectively collected data on ICU admissions. Setting. Forty Dutch ICUs. Patients: A data set from a national registry of 86,427 patients from January 2002 to October 2006. Interventions. The league tables associated with the different models were compared to evaluate their agreement. Bootstrapping was used to quantify the uncertainty in the ranks for ICUs. First, for each ICU the median rank and its 95% confidence interval were identified for each model. Then, for a given pair of models, for each ICU the median difference in rank and its associated 95% confidence interval were computed. A difference in rank for an ICU for a given pair of models was considered relevant if it was statistically significant and if one of the models would categorize this ICU as a performance outlier (excellent performer or very poor performer) while the other did not. Measurements and Main Results. For 20 ICUs, there was a significant difference in rank (2-19 positions) between one or more-pairs of models. Three ICUs were rated as performance outliers by one of the models, while the other excluded this possibility with 95% certainty. Furthermore, for ten ICUs, one or more pairs of models classified these ICUs as performance outliers while the other model did not do so with certainty. Regarding the agreement between the original models and their customized versions, in all cases the median change in rank was three positions or less and the models fully agreed with respect to which ICUs should be classified as performance outliers. Conclusions. Institutional comparison based on case-mix adjusted league tables is sensitive to the choice of prognostic model but not to customization of these models. League tables should always display the uncertainty associated with institutional ranks.