Interrater agreement reconsidered:: An alternative to the rwg indices

For continuous constructs, the most frequently used index of interrater agreement (r(wg(1))) can be problematic. Typically, r(wg(1)) is estimated with the assumption that a uniform distribution represents no agreement. The authors review the limitations of this uniform null r(wg(1)) index and discuss alternative methods for measuring interrater agreement. A new interrater agreement statistic, a(wg(1)), is proposed. The authors derive the a(wg(1)) statistic and demonstrate that a(wg(1)) is an analogue to Cohen's kappa, an interrater agreement index for nominal data. A comparison is made between agreement estimates based on the uniform r(wg(1)) and a(wg(1)), and issues such as minimum sample size andpractical significance levels are discussed. The authors close with recommendations regarding the use of r(wg(1))/r(wg(J)) when a uniform null is assumed, indices that do not assume a uniform null, a(wg(1))/a(wg(J)) indices, and generalizability estimates of interrater agreement.