Relevance thresholds: a multi-stage predictive model of how users evaluate information

This investigation examines end-user judgment and evaluation behavior during information retrieval (IR) system interactions and extends previous research surrounding relevance as a key construct for representing the value end-users ascribe to items retrieved from IR systems. A self-reporting instrument collected evaluative responses from 32 end-users related to 1432 retrieved items in relation to five characteristics of each item: (1) whether it was on topic, (2) whether it was meaningful to the user, (3) whether it was useful in relation to the problem at hand, (4) whether the IR system returned the information in the right form or format, and (5) whether the information retrieved allowed the user to take further action on it. The manner in which these characteristics of the retrieved items were considered, differentiated and aggregated were examined in relation to the region of relevance attributed to those items by the users. The nominal nature of the data collected led to non-parametric statistical analyses that indicated that end-user evaluation of retrieved items to resolve an information problem at hand is most likely a multi-stage process. While end-users may differ in their judgments and evaluations of retrieved items, they appear to make those decisions, by using somewhat consistent heuristic approaches that point to a predictive multi-stage model of relevance thresholds that exist on a continuum from topic to meaning (pertinence) to functionality (use): (C) 2002 Elsevier Science Ltd. All rights reserved.