Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials

Context Specific features of trial quality maybe associated with exaggeration or shrinking of the observed treatment effect in randomized studies. Therefore, assessment of trial quality is often used in meta-analysis. However, the degree to which specific quality measures are associated with treatment effects has not been well established across a broad range of clinical areas. Objective To determine if quality measures are associated with treatment effect size in randomized controlled trials (RCTs). Design Quality measures from published quality assessment scales were evaluated in RCTs included in meta-analyses from 4 medical areas (cardiovascular disease, infectious disease, pediatrics, and surgery). Included meta-analyses incorporated at least 6 RCTs, examined dichotomous outcomes, and demonstrated significant between-study heterogeneity in the odds ratio (OR) scale. Main Outcome Measures Relative ORs comparing overall treatment effect (summary OR) of high- vs low-quality studies, as determined by each quality measure, with relative ORs less than 1 indicating larger treatment effect in low-quality studies. Results Twenty-four quality measures were analyzed for 276 RCTs from 26 meta-analyses. Relative ORs of high- vs low-quality studies for these quality measures ranged from 0.83 to 1.26; none was statistically significantly associated with treatment effect. The proportion of studies fulfilling specific quality measures varied widely in the 4 medical areas. In analyses limited to specific medical areas, placebo control, multicenter studies, study country, caregiver blinding, and statistical methods were significantly associated with treatment effect on 7 occasions. These relative ORs ranged from 0.40 to 1.74. However, the directions of these associations were not consistent. Conclusions Individual quality measures are not reliably associated with the strength of treatment effect across studies and medical areas. Although use of specific quality measures may be appropriate in specific well-defined areas in which there is pertinent evidence, findings of associations with treatment effect cannot be generalized to all clinical areas or meta-analyses.