Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings

In patients with community-acquired pneumonia (CAP) prediction rules based oil individual predicted mortalities are frequently used to support decision-making for in-patient vs. outpatient management. We studied the accuracy and the need for recalibration of three risk prediction scores in a tertiary-care University hospital emergency-department setting in Switzerland. We pooled data from patients with CAP enrolled in two randomized controlled trials. We compared expected mortality from the original pneumonia severity index (PSI), CURB65 and CRB65 scores against observed mortality (calibration) and recalibrated the scores by fitting the intercept alpha and the calibration slope beta from our calibration model. Each of the original models underestimated the observed 30-day mortality of 11%, in 371 patients admitted to the emergency department with CAP (8.4%, 5.5% and 5-0% for the PSI, CURB65 and CRB65 scores, respectively). In particular, we observed a relevant mortality within the low risk classes of the original models (2.6%, 5.3%, and 3.7% for PSI classes I-III, CURB65 classes 0-1, and CRB65 class 0, respectively). Recalibration of the original risk models corrected the miscalibration. After recalibration, however, only PSI class I was sensitive enough to identify patients with a low risk (i.e. < 1 %) for mortality suitable for outpatient management. In our tertiary-care setting with mostly referred in-patients, CAP risk scores substantially underestimated observed mortalities misclassifying patients with relevant risks of death suitable for outpatient management. Prior to the implementation of CAP risk scores in the clinical setting, the need for recalibration and the accuracy of low-risk re-classification should be studied in order to adhere with discharge guidelines and guarantee patients' safety.