Bias in meta-analysis due to outcome variable selection within studies

Although bias in meta-analysis arising from selective publication has been studied, within-study selection has received little attention. Chronic diseases often have several possible outcome variables. Methods based on the size of the effect allow results from studies with different outcomes to be combined. However, the possibility of selective reporting of outcomes must be considered. The effect of selective reporting on estimates of the size of the effect and significance levels is presented, and sensitivity analyses are suggested. Substantial publication bias could arise from multiple testing of outcomes in a study, followed by selective reporting. Two meta-analyses, on anthelminth therapy and a treatment for incontinence, are reassessed allowing for within-study selection, as it is clear that more outcomes had been measured than were reported. The sensitivity analyses show that the robustness of the anthelminth results is dependent on what assumption one makes about the reporting strategy for the largest trial. The possible influence of correlation between within-child measurements was such that the conclusions could easily be reversed. The effect of a mild assumption on within-trial selection alone could alter general recommendations about the treatment for incontinence.