Data mining and KDD: Promise and challenges

Databases are growing in size to a stage where traditional techniques for analysis and visualization of the data are breaking down. Data mining and knowledge discovery in databases (KDD) are concerned with extracting models and patterns of interest from large databases. Data mining techniques have their origins in methods from statistics, pattern recognition, databases, artificial intelligence, high performance and parallel computing, and visualization. In this article, we provide an overview of this growing multi-disciplinary research area, outline the basic techniques, and provide brief coverage of how they are used in some applications. We discuss the role of high performance and parallel computing in data mining problems, and we provide a brief overview of a few applications in science data analysis. We conclude by listing challenges and opportunites for future research.