Prediction rules for complications in coronary bypass surgery - A comparison and methodological critique

BACKGROUND. Clinical prediction rules have been developed that use preoperative information to stratify patients according to risk of complications after cardiac surgery. OBJECTIVES. TO assess the methodological standards and performance of 7 models. PARTICIPANTS. The validation portion of the Quality Measurement and Management Initiative (QMMI) cohort included a random sample of all adult patients (n = 3,261) who underwent coronary artery bypass grafting (CABG) surgery not involving valvular or other concomitant procedures at 12 medical centers from August 1993 to October 1995. OUTCOME MEASURES. Methodological standards used for model comparison were adapted from published criteria. Model performance was assessed by receiver-operating characteristic (ROC) analysis, and calibration was evaluated with the Hosmer-Lemeshow (HL) statistic and observed-expected plots. METHODS. We performed cross-validation by applying the published criteria for the development of each model to the validation subset of the QMMI cohort and by assessing the performance of each model in discriminating outcomes. RESULTS. Wide variations existed in the methodologies used to develop and validate the 5 additive scores evaluated. Cross-validation of all 5 additive scores revealed degradation in their abilities to discriminate outcomes. The 2 logistic models examined performed similarly to the additive scores examined in predicting mortality. CONCLUSIONS. Substantial variation existed both in the methodologies used to develop models and in the ability of the models to predict outcomes. Models developed at single institutions or using fewer patients may be less generalizable when applied to diverse clinical settings. Additive and logistic regression models performed similarly, as assessed by ROC and HL analyses.