On effective conceptual indexing and similarity search in text data

被引:13
作者
Aggarwal, CC [1 ]
Yu, PS [1 ]
机构
[1] IBM Corp, Thomas J Watson Res Ctr, Yorktown Hts, NY 10598 USA
来源
2001 IEEE INTERNATIONAL CONFERENCE ON DATA MINING, PROCEEDINGS | 2001年
关键词
D O I
10.1109/ICDM.2001.989494
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Similarity search in text has proven to be an interesting problem from the qualitative perspective because of inherent redundancies and ambiguities in textual descriptions. The methods used in search engines in order to retrieve documents most similar to user-defined sets of keywords are not applicable to targets which are medium to large size documents, because of even greater noise effects stemming from the presence of a large number of words unrelated to the overall topic in the document. The inverted representation is the dominant method for indexing text, but it is not as suitable for document-to-document similarity search, as for short user-queries. One way of improving the quality, of similarity search is Latent Semantic Indexing (LSI), which maps the documents from the original set of words to a concept space. Unfortunately, LSI maps the data into a domain in which it is not possible to provide effective indexing techniques. In this paper, we investigate new ways of providing conceptual search among documents by creating a representation in terms of conceptual word-chains. This technique also allows effective indexing techniques so that similarity queries can be performed on large collections of documents by accessing a small amount of data. We demonstrate that our scheme outperforms standard textual similarity search on the inverted representation both in terms of quality and search efficiency.
引用
收藏
页码:3 / 10
页数:8
相关论文
共 12 条
  • [1] Aggarwal C. C., 2001, ACM KDD C
  • [2] AGGARWAL CC, 1999, ACM SIGKDD C
  • [3] AGGARWAL CC, 2001, ACM PODS C
  • [4] ANICK P, 1997, ACM SIGIR C
  • [5] Baker L., 1998, ACM SIGIR C
  • [6] DUMAIS S, 1988, ACM SIGCHI C
  • [7] FALOUTSOS C, 1995, ACM COMPUTING SURVEY, V17
  • [8] HINNEBURG A, 2001, P VLDB C
  • [9] KARYPIS G, 2000, CIKM C
  • [10] KLEINBERG J, 1999, ACM PODS C