The effect of collapsing multinomial data when assessing agreement

Background In epidemiological studies researchers often depend on proxies to obtain information when primary subjects are unavailable. However, relatively few studies have performed formal statistical inference to assess agreement among proxy informants and primary study subjects. In this paper, we consider inference procedures for studies of interobserver agreement characterized by two raters and three or more outcome categories. Of particular interest is the consequence of dichotomizing such data on the expected confidence interval width for the kappa coefficient. The effect of dichotomization on sample size requirements for testing hypotheses concerning kappa is also evaluated. Methods Simulation studies were used to compare coverage levels and widths for constructing confidence intervals. Sample size requirements were compared for multinomial and dichotomous data. We illustrate our results using a published data set on drinking habits that assesses agreement among primary and proxy respondents. Results Our results show that when multinomial data are treated as dichotomous, not only do the expected confidence interval widths became greater, but the penalty in terms of larger sample size requirements for hypothesis testing can be severe. Conclusion We conclude that there are clear advantages in preserving multinomial data on the original scale rather than collapsing the data into a binary trait.