STATISTICALLY CONTROLLED ACTIVATION WEIGHT INITIALIZATION (SCAWI)

An optimum weight initialization which strongly improves the performance of the BP algorithm is suggested. By statistical analysis, the scale factor, R (which is proportional to the maximum magnitude of the weights), is obtained as a function of the paralyzed neuron percentage (PNP). Also, by computer simulation, the performances on the convergence speed have been related to PNP. An optimum range for R is shown to exist in order to minimize the time needed to reach the minimum of the cost function. The analysis is carried on by properly defining normalization factors, which leads to a distribution of the activations independent of the neurons, and to a single nondimensional quantity, R, whose value may be quickly found by computer simulation.