Measuring interrater agreement for ratings of a single target

Researchers assessing interrater agreement for ratings of a single target have increasingly used the r(WG(j)) index, but have found it can display irregular behavior. Mathematical analyses show this problem arises from the use of random response, operationalized by the variance of a uniform distribution (s(EU)(2)), for the baseline of comparison. These analyses suggest that researchers should continue to use r(WG)(j) as a summary measure of interrater agreement, but should use maximum dissensus as a reference distribution for computing r(WG)(j). Although values of s(EU)(2) can be descriptively misleading, they provide an important inferential baseline. Thus, s(EU)(2) should be used in computing chi(2) tests Of the departure of the observed response variance from random responding. Researchers should also examine interrater agreement as a theoretical variable in its own right, investigating the causes and consequences of rater dissensus.