Undue reliance on I2 in assessing heterogeneity may mislead

Background: The heterogeneity statistic I-2, interpreted as the percentage of variability due to heterogeneity between studies rather than sampling error, depends on precision, that is, the size of the studies included. Methods: Based on a real meta-analysis, we simulate artificially 'inflating' the sample size under the random effects model. For a given inflation factor M = 1, 2, 3,... and for each trial i, we create a M-inflated trial by drawing a treatment effect estimate from the random effects model, using S-2(i)/M as within-trial sampling variance. Results: As precision increases, while estimates of the heterogeneity variance tau(2) remain unchanged on average, estimates of I2 increase rapidly to nearly 100%. A similar phenomenon is apparent in a sample of 157 meta-analyses. Conclusion: When deciding whether or not to pool treatment estimates in a meta-analysis, the yard-stick should be the clinical relevance of any heterogeneity present. tau(2), rather than I-2, is the appropriate measure for this purpose.