Small studies are more heterogeneous than large ones: a meta-meta-analysis

Objectives: Between-study heterogeneity plays an important role in random-effects models for meta-analysis. Most clinical trials are small, and small trials are often associated with larger effect sizes. We empirically evaluated whether there is also a relationship between trial size and heterogeneity (tau). Study Design and Setting: We selected the first meta-analysis per intervention review of the Cochrane Database of Systematic Reviews Issues 2009-2013 with a dichotomous (n = 2,009) or continuous (n = 1,254) outcome. The association between estimated tau and trial size was evaluated across meta-analyses using regression and within meta-analyses using a Bayesian approach. Small trials were predefined as those having standard errors (SEs) over 0.2 standardized effects. Results: Most meta-analyses were based on few (median 4) trials. Within the same meta-analysis, the small study tau(2)(S) was larger than the large-study tau(2)(L) [average ratio 2.11; 95% credible interval (1.05, 3.87) for dichotomous and 3.11 (2.00, 4.78) for continuous meta-analyses]. The imprecision of tau(S) was larger than of tau(L): median SE 0.39 vs. 0.20 for dichotomous and 0.22 vs. 0.13 for continuous small-study and large-study meta-analyses. Conclusion: Heterogeneity between small studies is larger than between larger studies. The large imprecision with which tau is estimated in a typical small-studies' meta-analysis is another reason for concern, and sensitivity analyses are recommended. (C) 2015 Elsevier Inc. All rights reserved.