Comparison of visual inspection and statistical analysis of single-subject data in rehabilitation research

Single-subject designs are being advocated to conduct outcome research in rehabilitation environments. The methods provide an alternative to traditional designs based on statistical comparisons across groups. Data analysis in single subject research does not rely on statistical hypothesis testing of responses collected from a sample of subjects. Instead, visual inspection of patient responses graphed over time is the usual method of data analysis in single-subject research. This study examined the agreement between visual analysis and statistical tests of single-subject data for 42 hypothetical single-subject graphs. Specially constructed graphs allowed the systematic manipulation of different treatment effect sizes across a commonly used single-subject design. Thirty-two rehabilitation and health care providers rated each of the 42 graphs to determine whether a clinically significant treatment effect existed across the phases of the designs. Data analysis focused on two questions: (1) How much agreement was there between visual judgments and the results of statistical tests? and (2) What level of treatment effect was required to produce a finding of visual versus statistical significance? The agreement between visual analysis and statistical significance was high (86%). The sensitivity of visual inferences compared with statistical test results was 0.84, specificity was 0.88, and positive predictive value was 0.91. Both visual and statistical procedures were sensitive to medium and large treatment effects in the 42 single-subject graphs examined in this study.