TOLERANCE TO ANALOG HARDWARE OF ON-CHIP LEARNING IN BACKPROPAGATION NETWORKS

When training an artificial neural network using the popular backpropagation algorithm, implementation in dedicated analog hardware offers an attractive alternative for reasons of speed, compactness, and the lack of limited resolution in computation that is present in digital hardware, Despite these advantages, until now the use of analog hardware in backpropagation networks has been primarily limited to the forward computation, The learning computations are usually performed off-chip, because it is not immediately evident that the nonidealities inherent in analog hardware allow adequate learning to take place, In this paper we present results of simulations performed assuming both forward and backward computation are done on-chip using analog components, Aspects of analog hardware studied are component variability (variability in multiplier gains and zero offsets),limited voltage ranges, components (multipliers) that only approximate the computations in the backpropagation algorithm, and capacitive weight decay, It is shown that backpropagation networks can learn to compensate for all these shortcomings of analog circuits except for zero offsets, and the latter are correctable with minor circuit complications, Variability in multiplier gains is not a problem, and learning is still possible despite limited voltage ranges and function approximations, Fixed component variation from fabrication is shown to be less detrimental to learning than component variation due to noise, Weight decay is tolerable provided it is sufficiently:small, which implies frequent refreshing by rehearsal on the training data or modest cooling of the circuits, The former approach allows for learning nonstationary problem sets.