Measuring diagnostic agreement

Diagnostic agreement tests the reliability and concordance of diagnostic systems. the introduction of measures of agreement with reputations for baserate independence (e.g., Yule's Y and Q), and new studies occasioned by the publication of the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; American Psychiatric Association, 1994) and the International Classification of Diseases - 10 (ICD-10; World Health Organization, 1992) make it necessary to study the relationship of illness baserates to measures of agreement. Testing diagnostic concordance for diagnoses of drug dependence from the third edition of the DSM (American Psychiatric Association, 1980) versus DSM-IV diagnoses of drug dependence under 3 baserate conditions, it was found that Yule's Y and a proved as vulnerable to differences in baserates as kappa or percent agreement and that specificity covaried with baserate rather than being fixed, as most theoretical discussions assume. The uncritical use of Y and Q, therefore, is likely to lead to optimistic interpretations of agreement. Kappa should be preferred for most purposes, although an adjustment to the computational formulas for Y and a is presented that can diminish their positive bias.