STATISTICAL-ENSEMBLE THEORY OF REDUNDANCY REDUCTION AND THE DUALITY BETWEEN UNSUPERVISED AND SUPERVISED NEURAL LEARNING

The aim of this paper is twofold. First, we derive a statistical-mechanics-based model of unsupervised learning defined by redundancy reduction between the output components of neural nets and entropy conservation from inputs to outputs. We obtain an approximate expression for the probability distribution of the output components for a new data point, which is essentially determined by the probability distribution given by the best network of neural ensembles and by the square root of the ratio between the determinants of the Fisher information without and with the new point. Second, we pose the problem of supervised learning as an unsupervised one. The ensemble theory derived for unsupervised learning results in one for supervised learning by using the ensemble theory based on the maximum-likelihood principle. An upper bound for the prediction probability of a new point not included in the training data is derived. This upper bound is essentially given by the ratio between the Fisher information, determined for the training sets without and with the inclusion of the new point. This upper bound may be used as a mechanism to decide actively on the novelty of new data (mechanism of query learning). An illustrative example is given for the case where the training error possesses a Gaussian distribution.