THE AUTOMATIC IDENTIFICATION OF STOP WORDS

A stop word may be identified as a word that has the same likelihood of occurring in those documents not relevant to a query as in those documents relevant to the query. In this paper we show how the concept of relevance may be replaced by the condition of being highly rated by a similarity measure. Thus it becomes possible to identify the stop words in a collection by automated statistical testing. We describe the nature of the statistical test as it is realized with a vector retrieval methodology based on the cosine coefficient of document-document similarity. As an example, this technique is then applied to a large MEDLINE(R) subset in the area of biotechnology. The initial processing of this database involves a 310 word stop list of common non-content terms. Our technique is then applied and 75% of the remaining terms are identified as stop words. We compare retrieval with and without the removal of these stop words and find that of the top twenty documents retrieved in response to a random query document, seventeen of these are the same on the average for the two methods. We also examine the differences and conclude that where the user prefers one method over the other, the new method with the reduced term set is favored about three times out of four.