LOG-LINEAR MODELING OF PAIRWISE INTEROBSERVER AGREEMENT ON A CATEGORICAL SCALE

This article uses log-linear models to describe pairwise agreement among several raters who classify a sample on a subjective categorical scale. The models describe agreement structure simultaneously for second-order marginal tables of a multidimensional cross-classification of ratings. Practical difficulties arise in fitting the models, because models refer to pairwise marginal tables of a very large and sparse table. A standard analysis that treats the marginal tables as independent yields consistent estimates of model parameters, but not of the covariance matrix of the estimates. We estimate the covariance matrix using the jackknife. We apply the models to describe agreement between evaluations made by seven pathologists of carcinoma in situ of the uterine cervix, using a five-level ordinal scale. Previous analyses showed differences among the pathologists in their pairwise levels of agreement, but we observe near homogeneity in the dependence structure of their ratings.