How useful are observational reports in the evaluation of interventions for radiation morbidity? an analysis of formalin therapy for late radiation proctitis

Background and purpose: Evidence-based medicine requires the systematic and critical evaluation of published and unpublished trials. Problems arise when a clinical condition is relatively rare and the only available data relate to experiential knowledge. The way forward would be to recommend the development of good quality randomized controlled studies. Until then, we are left with the situation where some information exists, albeit in the form of case reports and small series. Should this information be used and what features would determine its strength? Methods: Using the example of formalin therapy in haemorrhagic radiation proctitis, a treatment for a rare condition, we were able to identify 16 published studies, 13 of which were retrospective and three of which were prospective. The quality of reporting detail was assessed by comparison to the features in a 'proposed minimum dataset' for an uncontrolled study addressing this topic. Results: The mean score for quality of reporting detail for these studies was 50.6% (range 25-70%). Earlier studies reported a significantly higher response rate than subsequent studies and although there was a tendency for smaller studies to report higher response rates, this was not significant. The score for detail of reporting did not improve with year of publication and the correlation between the size of the study and the detail of reporting was not statistically significant. Conclusions: The information presented is of exceedingly variable quality. If these studies are to be used, where insufficient controlled trials are available, they should be scored for methodology, and these scores used to assist interpretation of results. This would be facilitated if an accepted reporting format including specific criteria was available. (C) 2002 Elsevier Science Ireland Ltd. All rights reserved.