Assessment of observer performance in a subjective scoring system: visual classification of the gait of cows

As with any measurement procedure, the performance of a subjective classification procedure must be evaluated. Observers have to be trained and their performance has to be assessed, preferably on a regular basis, to guarantee sufficient consistency and accuracy of classification results. The current paper is a study of observer performance where observers were asked to classify the gait of cows from video recordings. Gait was classified in nine ordered categories (ranging from 1 = normal gait to 9 = severely abnormal gait) and also as a continuous fraction by putting a mark on a paper strip (the left end corresponding to 0 = normal gait and the right end to I = severely abnormal gait). The use of statistical models and methodology for analysis of these visual scores is demonstrated and discussed. Observers were assessed by comparing their classification results with the results of an expert. Models and methodology take proper account of typical features of the data, i.e. the fact that data are discrete scores or continuous scores with an upper and lower bound, the variance heterogeneity and non-linearity of model terms that arises from this, and the dependence between repeated classifications of videos of the same cow. Results of the analyses are summarized in simple tables and plots. These are useful tools to indicate possible flaws in judgement of an observer, that may be corrected by further training. When a high standard is developed, which usually takes the form of the opinion of one or more experts, this methodology can be applied prior to any experiment where responses are ordered subjective scores.