A DISAGREEMENT ABOUT WITHIN-GROUP AGREEMENT - DISENTANGLING ISSUES OF CONSISTENCY VERSUS CONSENSUS

James, Demaree, and Wolf (1984) developed an index, r(WG), for assessing within-group agreement appropriate when only a single target is rated. The assessment of interperceiver agreement in such situations is of particular relevance to the composition model for climate. Schmidt and Hunter (1989) have criticized the conceptual foundation of r(WG) because it is not consistent with the classical model of reliability. They proposed an alternative approach, the use of the rating standard deviation (SD(x)), the standard error of the rating mean (SE(M)), and the associated confidence intervals for SE(M) to index interrater agreement. This comment argues that the critique of r(WG) did not clearly distinguish the concepts of interrater consensus (i.e., agreement) and interrater consistency (i.e., reliability). When the distinction between agreement and reliability is carefully drawn, the critique of r(WG) is shown to divert attention from more critical problems in the assessment of agreement. A comparison demonstrates that the approach for assessing within-group agreement proposed by Schmidt and Hunter (1989) suffers from several limitations. The comment concludes that r(WG) should not be used as an index of interrater reliability but, within certain bounds, it is suitable as an index of within-group interrater agreement and that SD(x) and SE(M) are not acceptable substitutes for extant indexes of interrater agreement.