Publication bias and meta-analyses - A practical example

Objectives: Publication bias is widely appreciated, but considerable time and effort are needed to locate and obtain data from unpublished randomized controlled trials (RCTs), those published in non-English language journals or those reported in the gray literature; for this publication, we will call this collection of trials the "gray+literature." However, excluding such trials from systematic reviews could introduce bias and give rise to misleading conclusions. Methods: We aimed to explore and quantify the impact of inclusion of gray+ literature on the results of all completed individual patient data (lPD) reviews coordinated by our group (13 meta-analyses). For each lPD review, results were calculated for RCTs fully published in English language journals and RCTs fully published in English language journals and the gray+ literature. Results: The IPD meta-analyses based only on RCTs that were fully published in English language journals tended to give more favorable results than those that included RCTs from the gray+ literature. Although in most cases the addition of gray+ data gave less encouraging results, moving the estimated treatment effect toward a null result, the direction of effect was not always predictable. Conclusions: We recommend that all systematic reviews should at least attempt to identify trials reported in the gray+ literature and, where possible, obtain data from them.