Mining data to find subsets of high activity

Many data mining problems in biometrics research are concerned with trying to identify the characteristics of a subset of cases that responds substantially differently from the rest of the cases. For example, when studying the relationship between a response variable Y and a set of predictor variables, it is often of interest to determine what ranges of values of the predictor variables are associated with a high likelihood of Y = 1 (if Y is a Bernoulli variable) or with high values of Y (if Y is a continuous variable). We describe a criterion (H) and a recursive partitioning method (ARF) that directly addresses this question. A computational algorithm that makes ARF feasible for use even with very large datasets is presented. The basic version of ARF can be generalized to the case of multiple response variables, Y1,...,Y-t and other settings. We illustrate the effectiveness of ARF by mining a structure activity database, a hospital database, and some other real and simulated datasets. We conclude by proposing a basic paradigm for data mining. (C) 2003 Published by Elsevier B.V.