EVENT-DEPENDENT CONTROL OF NOISE ENHANCES LEARNING IN NEURAL NETWORKS

We have devised noise-control algorithms, using biological adaptation as an analogy, for application to response optimization in adaptive systems generally. The present paper illustrates one of these algorithms by showing its effects on increasing the rate of learning in neural networks. Optimization procedures usually employ simulated annealing by which noise is systematically decreased at a constant rate. Our methods are time-invariant, and control the level of injected noise solely through the response of the system. Such time-invariant noise algorithms (TINA) may be more applicable than annealing to adaptive systems that must respond to unpredictable environments, and may find analogy in brain function. Both TINA and annealing have surprising properties of a new form of generalization in which networks that have been trained in the presence of noise are able to exhibit enhanced rates of learning in a subsequent learning task when no noise is present. We use special features of the geometry of error-surfaces, depicting the error as a function of changes in synaptic weights, to discuss the effect of noise in enhancing the rate of learning, and to compare learning strategies available to networks exposed to the different training procedures. The applicability of the findings to biological systems is discussed.