CONVERGENCE OF LEARNING ALGORITHMS WITH CONSTANT LEARNING RATES

We investigate the behavior or neural network learning algorithms with a small, constant learning rate, is-an-element-of, in stationary, random input environments. It is rigorously established that the sequence of weight estimates can be approximated by a certain ordinary differential equation, in the sense of weak convergence of random processes as is-an-element-of tends to zero. As applications, back-propagation in feedforward architectures and some feature extraction algorithms are studied in more detail.