UNSUPERVISED MUTUAL INFORMATION CRITERION FOR ELIMINATION OF OVERTRAINING IN SUPERVISED MULTILAYER NETWORKS

Controlling the network complexity in order to prevent overfitting is one of the major problems encountered when using neural network models to extract the structure from small data sets. In this paper we present a network architecture designed for use with a cost function that includes a novel complexity penalty term. In this architecture the outputs of the hidden units are strictly positive and sum to one, and their outputs are defined as the probability that the actual input belongs to a certain class formed during learning. The penalty term expresses the mutual information between the inputs and the extracted classes. This measure effectively describes the network complexity with respect to the given data in an unsupervised fashion. The efficiency of this architechture/penalty-term when combined with back-propagation training, is demonstrated on a real world economic time series forecasting problem. The model was also applied to the bench-mark sunspot data and to a synthetic data set from the statistics community.