RELIABILITY AND EXPECTED LOSS - A UNIFYING PRINCIPLE

被引:14
作者
COOIL, B
RUST, RT
机构
[1] Owen Graduate School of Management, Vanderbilt University, Nashville, 37203, TN
关键词
ALPHA; KAPPA; AGREEMENT; INTERCODER RELIABILITY; DECISION RULE; GENERALIZABILITY THEORY; TEST THEORY;
D O I
10.1007/BF02295184
中图分类号
O1 [数学];
学科分类号
0701 ; 070101 ;
摘要
We provide a unified, theoretical basis on which measures of data reliability may be derived or evaluated, for both quantitative and qualitative data. This approach evaluates reliability as the ''proportional reduction in loss'' (PRL) that is attained in a sample by an optimal estimator. The resulting measure is between 0 and 1, linearly related to expected loss, and provides a direct way of contrasting the measured reliability in the sample with the least reliable and most reliable data-generating cases. The PRL measure is a generalization of many of the commonly-used reliability measures. We show how the quantitative measures from generalizability theory can be derived as PRL measures (including Cronbach's alpha and measures proposed by Winer). For categorical data, we develop a new measure for the general case in which each of N judges assigns a subject to one of K categories and show that it is equivalent to a measure proposed by Perreault and Leigh for the case where N is 2.
引用
收藏
页码:203 / 216
页数:14
相关论文
共 17 条
[1]  
[Anonymous], 1971, STAT PRINCIPLES EXPT
[2]   COEFFICIENT KAPPA - SOME USES, MISUSES, AND ALTERNATIVES [J].
BRENNAN, RL ;
PREDIGER, DJ .
EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1981, 41 (03) :687-699
[3]  
BRENNAN RL, 1983, ELEMENTS GENERALIZAB
[5]   A COEFFICIENT OF AGREEMENT FOR NOMINAL SCALES [J].
COHEN, J .
EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1960, 20 (01) :37-46
[6]   INTEGRATION AND GENERALIZATION OF KAPPAS FOR MULTIPLE RATERS [J].
CONGER, AJ .
PSYCHOLOGICAL BULLETIN, 1980, 88 (02) :322-328
[7]   CRITERIA FOR MEASURES OF ASSOCIATION [J].
COSTNER, HL .
AMERICAN SOCIOLOGICAL REVIEW, 1965, 30 (03) :341-353
[8]  
Cronbach LJ, 1951, PSYCHOMETRIKA, V16, P297
[9]  
CRONBACH LJ, 1972, DEPENDABILITY BEHAVI
[10]  
Hildebrand D.K., 1977, PREDICTION ANAL CROS