AN EXACTLY SOLVABLE MODEL OF UNSUPERVISED LEARNING

A model for unsupervised learning from N-dimensional data is studied. Random training examples are drawn such that the distribution of their overlaps with a vector B is-an-element-of R(N) is a mixture of two Gaussians of unit width and a separation rho. A student vector is generated by an on-line algorithm, using each example only once. The evolution of its overlap with B can be calculated exactly in the themodynamic limit N --> infinity. As a specific example, a learning algorithm closely related to Oja's rule is investigated. Its dynamics and approach to the stationary solution are solved for both a constant and an optimally chosen time-dependent learning rate. For the latter, the limits of small and infinitely large separation rho of the peaks are considered. In both limits the analysis suggests the use of an asymptotic (1/p)-decay for the learning rate, where p is the number of training examples. In the large-separation limit, the typical number of examples needed for successful learning is found to be (p/N) is-proportional-to rho-2, which coincides with a recent result for supervised learning from Gaussian mixtures.