An assessment of observer bias in the shunt design trial

Introduction: An international multicenter randomized trial comparing standard pressure differential valves, Orbis Sigma valves and PS Medical Delta valves, for children with newly diagnosed hydrocephalus failed to show a difference in the time to first shunt failure (power 80%). In order to avoid observer bias, the primary end-point, shunt failure, was defined by detailed clinical and radiological criteria. Surgeons notified the data center when their patient reached endpoint. Their decision was blindly reviewed by a centralized adjudication process. The current analysis asks: (1)Did the surgeons and the blinded adjudication process agree on the presence of shunt failure? (2) Do the shunt survival curves change when the surgeons' assessment of outcome is used? Methods: The two methods of outcome assessment (surgeons' decision and adjudication process) were compared with a kappa statistic for the presence of shunt failure and a log rank statistic for time to shunt failure. Results: The surgeon and the adjudication process agreed on the presence of the primary outcome in 313/344 patients in the trial (kappa = 0.81). The 31 cases of disagreement were evenly distributed among the three shunts. The survival curves generated from the surgeons' decision were not significantly different from those generated by the adjudication process (log rank = 0.70, p = 0.37) and did not alter the study conclusions. Discussion: Observer bias in the assessment of outcome is always a concern in an unblinded clinical trial such as this one. We did not detect an observer bias of sufficient magnitude to alter the study conclusions.