Decision support issues using a physiology based score

Objective: As physiology based assessments of mortality risk become more accurate, their potential utility in clinical decision support and resource rationing decisions increases. Before these prediction models can be used, however, their performance must be statistically evaluated and interpreted in a clinical context. We examine the issues of confidence intervals (as estimates of survival ranges) and confidence levels (as estimates of clinical certainty) by applying Pediatric Risk of Mortality III (PRISM III) in two scenarios: (1) survival prediction for individual patients and (2) resource rationing. Design: A non-concurrent cohort study. Setting: 32 pediatric intensive care units (PICUs). Patients: 10608 consecutive patients (571 deaths). Interventions: None. Measurements and results: For the individual patient application, we investigated the observed survival rates for patients with low survival predictions and the confidence intervals associated with these predictions. For the resource rationing application, we investigated the maximum error rate of a policy which would limit therapy for patients with scores exceeding a very high threshold. For both applications, we also investigated how the confidence intervals change as the confidence levels change. The observed survival in the PRISM III groups > 28, > 35, and > 42 were 6.3, 5.3, and 0 %, with 95 % upper confidence interval bounds of 10.5, 13.0, and 13.3 %, respectively. Changing the confidence level altered the survival range by more than 300 % in the highest risk group, indicating the importance of clinical certainty provisions in prognostic estimates. The maximum error rates for resource allocation decisions were low (e. g., 29 per 100 000 at a 95 % certainty level), equivalent to many of the risks of daily living. Changes in confidence level had relatively little effect on this result. Conclusions: Predictions for an individual patient's risk of death with a high PRISM score are statistically not precise by virtue of the small number of patients in these groups and the resulting wide confidence intervals. Clinical certainty (confidence level) issues substantially influence outcome ranges for individual patients, directly affecting the utility of scores for individual patient use. However, sample sizes are sufficient for rationing decisions for many groups with higher certainty levels. Before there can be widespread acceptance of this type of decision support, physicians and families must confront what they believe is adequate certainty.