BACK PROPAGATION LEARNING WITH TRINARY QUANTIZATION OF WEIGHT UPDATES

We have studied a simple variant of back propagation learning in networks with three-layer feedforward architecture, in which weight updates are constrained to assume one of only three values: an increment, a decrement of the same magnitude, or zero. An important motivation for this study is the potential for parallel implementation of learning rules with such coarsely quantized parameter changes in analog integrated circuitry. Trials performed on three small benchmark problems demonstrate that back propagation with this trinary update quantization can converge on satisfactory solutions of each of these problems, and within the range of parameters we used requires three to 10 times fewer iterations than a comparable version of standard back propagation. We present some evidence suggesting that this difference is due at least in part to the scaling imposed by the quantization scheme upon the vector of weight and bias updates. Tolerance of the final networks to parameter errors or faults in the form of random, normally distributed variations in the weight and bias values was examined and found to compare favorably with that of networks trained with standard back propagation.