Evaluation of two outcome prediction models on an independent database

Objective: To evaluate the performance of the New Simplified Acute Physiology Score (SAPS II) and the admission Mortality Probability Model (MPM0) in a large independent database, using formal statistical assessment. Design: Analysis of the database of a multicenter, multinational, prospective cohort study, EURICUS-I. Setting: Eighty nine intensive care units (ICUs) from 13 European areas. Patients: Data of 16,060 patients consecutively admitted to the participating ICUs were collected during a period of 4 months. Following the original SAPS II and MPM0 criteria, the analysis excluded: patients < 18 ys of age; readmissions; patients admitted with acute myocardial infarction; burns; and patients in the postoperative period after coronary artery bypass surgery. All patients with a length of stay < 8 hrs were excluded from the study to keep comparability between both systems. A total of 10,027 patients were analyzed. Interventions: Collection of the first 24 hrs' admission data necessary for the calculation of SAPS II and MPM0 and basic demographic statistics. Vital status at discharge from the hospital was registered. Measurements and Main Results: Despite having a good discriminative capability, as measured by the area under the receiver operating characteristic (ROC) curves (SAPS II: ROC = 0.822 +/- 0.005 SEM; MPM0: ROC = 0.785 +/- 0.006 SEM), both models presented poor calibration, with significant differences between observed and predicted mortality (Hosmer-Lemeshow goodness-of-fit tests H and C, p < .0001). Both SAPS II (predicted risk > 40%) and MPM0 (predicted risk > 30%) overestimated the risk of death. The evaluation of the uniformity of fit of SAPS II and MPM0 demonstrated large variations across the various subgroups of patients. Conclusions: The original SAPS II and MPM0 models did not accurately predict mortality on an independent large international multicenter ICU patient database. Results of studies utilizing general outcome prediction models without previous validation in the target population should be interpreted with prudence.