When to base clinical policies on observational versus randomized trial data

Physicians must decide when the evidence is sufficient to adopt a new clinical policy. Analysis of large clinical and administrative databases is becoming an important source of evidence for changing clinical policies. Because such analysis cannot control for the effects of all potential confounding variables, physicians risk drawing the wrong conclusion about the cause-and-effect relation between a change in clinical policy and outcomes. Randomized studies offer protection against drawing a conclusion that would lead to adoption of an inferior policy. However, a randomized study may be difficult to justify because of the extra costs of collecting data for a randomized study and concerns that a study will not directly benefit the patients enrolled in the study. This article reviews the advantages and disadvantages of basing clinical policy on analysis of large databases compared with conducting a randomized study. A technique is described and illustrated for accessing the potential costs and benefits of conducting such a study. This type of analysis formed the basis for a physician-managed health care organization deciding to sponsor a randomized study among patients with end-stage renal disease as part of a quality-improvement initiative.