A PROBABILISTIC LATENT CLASS MODEL FOR ASSESSING INTER-JUDGE RELIABILITY

Increasingly behavioral researchers are soliciting cognitive responses in addition to standard attitudinal measures when attempting to assess the effects of persuasive communications [in humans]. The coding of the elicited cognitive responses generally involves some sort of categorization, typically undertaken by independent judges, and the quality of the data is, to a large degree, evaluated in terms of some reliability coefficient which reflects the extent to which the independent judges agreed. This study presented and illustrated a probabilistic model for assessing inter-judge reliability. The proposed probabilistic model allows formal test statistics to be used to evaluate the extent and character of inter-judge reliability, estimate the assignment error rates and their standard errors and test for simultaneous agreement for more than 2 judges. The probabilistic model is operationalized in terms of restricted latent class models.