AN INFORMATION THEORETIC DESIGN AND TRAINING ALGORITHM FOR NEURAL NETWORKS

An algorithmic approach to designing feed-forward neural networks for pattern classification is presented. The technique computes a single hidden layer of nodes by adding one node at a time until the desired classification has been achieved. At each iteration, a node that maximizes an information theoretic measure is selected from a collection of candidates. The methodology is heuristic in nature intended to solve the NP-hard problem of constructing a neural network with a minimum number of nodes. Two strategies for computing a collection of candidate nodes are presented and some experimental results obtained by using the strategies are reported.