A STOCHASTIC VERSION OF THE DELTA-RULE

Self-organizing networks of neuron-like elements naturally lead to high-dimensional, nonlinear parameter spaces which prove difficult to search. Back-propagation is one of the simplest neural network/connectionist models that uses a gradient descent (delta rule) in a high multi-dimensional parameter space. Search in such a space is subject to many difficulties including minima that are locally stable but do not at the same time provide solutions. Search time can also be shown under relatively good conditions to scale poorly. Although gradient descent in error is attractive as a "weak" learning method it also seems to suffer from many search inefficiencies. A principle is proposed about the relation between constrained local noise injections and global search. The delta rule is modified to include synaptic noise in the transmission of information and modification of the connection strength. This stochastic version of the delta rule seems to promote escape from poor locally stable minima, and can improve convergence speed and likehood. © 1990.