LEARNING COEFFICIENT DEPENDENCE ON TRAINING SET SIZE

A rule for the selection of the learning coefficient, eta, for use in back propagation with batch training of neural networks is presented. The length of the error gradient is shown to increase as more training set examples are presented. This results in slow training or nonconvergence if eta is not decreased as the number of input examples increases. The effect of a momentum term is shown to allow a range of eta's to produce similar training rates. Two networks having identical topology are trained at different tasks, one with few training patterns (16) and one with many (192). Distinctly different values of eta are shown to produce good training for the two networks. We propose selecting eta-equal to 1.5 divided by the square root of the sum of the squares of the number of each input pattern type. Any group of similar inputs that map to identical outputs constitutes a pattern type. This rule produces a fixed value of eta that yields rapid training when coupled with a momentum coefficient of 0.9 for a wide variety of networks.