CLASSIFICATION TREES WITH NEURAL NETWORK FEATURE-EXTRACTION

Classification trees and multilayer feedforward neural networks are two popular approaches to the pattern recognition problem. In this paper, we propose the idea of using small multilayer nets at the decision nodes of a binary classification tree to extract nonlinear features. This approach exploits the power of tree classifiers to use appropriate local features at the different levels and nodes of the tree. The nets are trained and the tree is grown using a gradient-type learning algorithm in conjunction with a heuristic class aggregation algorithm in the multiclass case. The proposed method improves on standard classification tree design methods in that it generally produces trees with lower error rates and fewer nodes. It also provides a structured approach to neural network classifier design which reduces the problems associated with training large unstructured nets, and transfers the problem of selecting the size of the net to the simpler problem of finding a tree of the right size. A new, efficient tree pruning algorithm is proposed for this purpose. Trees constructed with the new method and the CART method [1] are compared on a waveform recognition problem and a handwritten character recognition problem. The new approach demonstrates significant decreases in error rate and tree size relative to CART, although the training time can be larger. It also yields comparable error rates and shorter training times than a large multilayer net trained with back-propagation on the same problems.