Overfitting cautious selection of classifier ensembles with genetic algorithms

Information fusion research has recently focused oil the characteristics of the decision profiles of ensemble members in order to optimize performance. These characteristics are particularly important in the selection of ensemble members. However, even though the control of overfitting is a challenge in machine learning problems, much less work has been devoted to the control of overfitting in selection tasks. The objectives of this paper are: (1) to show that overfitting call be detected at the selection stage: and (2) to present strategies to control overfitting. Decision trees and k nearest neighbors classifiers are used to create homogeneous ensembles, while single- and multi-objective genetic algorithms are employed as search algorithms at the selection sage. In this study, we use bagging and random subspace methods for ensemble generation. The classification error rate and a set of diversity measures are applied as search criteria. We show experimentally that the selection of classifier ensembles conducted by genetic algorithms is prone to overfitting, especially in the multi-objective case. In this study, the partial validation, backwarding and global validation strategies are tailored for classifier ensemble selection problem and compared. This comparison allows LIS to show that a global validation strategy should be applied to control overfilling in pattern recognition systems involving all ensemble member selection task. Furthermore, this study has helped LIS to establish that the global validation strategy call be used to measure the relationship between diversity and classification performance when diversity measure are employed as single-objective functions. (C) 2008 Elsevier B.V All rights reserved.