A universal method of information retrieval evaluation:: the "missing" link M and the universal IR surface

被引:10
作者
Egghe, L [1 ]
机构
[1] Limburgs Univ Ctr, B-3590 Diepenbeek, Belgium
关键词
universal IR surface; miss measure; precision; recall; fallout; silence; evaluation;
D O I
10.1016/S0306-4573(02)00094-8
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
The paper shows that the present evaluation methods in information retrieval (basically recall R and precision P and in some cases fallout F) lack universal comparability in the sense that their values depend on the generality of the IR problem. A solution is given by using all "parts" of the database, including the non-relevant documents and also the not-retrieved documents. It turns out that the solution is given by introducing the measure M being the fraction of the not-retrieved documents that are relevant (hence the "miss" measure). We prove that-independent of the IR problem or of the IR action-the quadruple (P. R; F; M) belongs to a universal IR surface, being the same for all IR-activities. This universality is then exploited by defining a new measure for evaluation in IR allowing for unbiased comparisons of all IR results. We also show that only using one, two or even three measures from the set {P, R, F, M} necessary leads to evaluation measures that are non-universal and hence not capable of comparing different IR situations. (C) 2003 Elsevier Ltd. All rights reserved.
引用
收藏
页码:21 / 30
页数:10
相关论文
共 17 条
[1]  
BOYCE BR, 1995, MEASUREMENT INFORMAT
[2]  
DOMINICH S, 2001, MATH FDN INFORMATION
[3]   Duality in information retrieval and the hypergeometric distribution [J].
Egghe, L ;
Rousseau, R .
JOURNAL OF DOCUMENTATION, 1997, 53 (05) :488-496
[4]  
EGGHE L, 2001, ELEMENTARY STAT EFFE
[5]  
FRANTS VI, 1997, AUTOMATED INFORMATIO
[6]  
GROSSMAN DA, 1988, INFORMATION RETRIEVA
[7]  
Heaps HS, 1978, INFORMATION RETRIEVA
[8]  
LEVERY F, 1968, DOCUMENTALISTE, V3, P3
[9]  
LOGUNOV AV, 1969, NAUCHNO TECKNICHES 2, V2, P5
[10]  
LOSEE RM, 1990, SCI INFORMATION