On-line training of recurrent neural networks with continuous topology adaptation

This paper presents a novel on-line procedure for training dynamic neural networks with input-output recurrences whose topology is continuously adjusted to the complexity of the target system dynamics. The latter is accomplished by changing the number of the elements of the network hidden layer whenever the existing topology cannot capture the dynamics presented by the new data. The training mechanism developed in this work is based on the suitably altered extended Kalman filter (EKF) algorithm which is simultaneously used for the network parameter adjustment and for its state estimation. The network itself consists of a single hidden layer with a Gaussian radial basis functions (GRBF's) and of a linear output layer. The choice of the GRBF is induced by the requirements of the online learning. The latter implies the network architecture which permits only local influence of the new data point in order not to forget the previously learned dynamics. The continuous topology adaptation is implemented in our algorithm to avoid memory and computational problems of using a regular grid of GRBF's which covers the network input space. Furthermore, we show that the resulting parameter increase can be handled ''smoothly'' without interfering with the already acquired information. In the case when the target system dynamics are changing over time, we show that a suitable forgetting factor can he used to ''unlearn'' the no-longer-relevant dynamics. The quality of the presented recurrent network training algorithm is demonstrated on the identification of nonlinear dynamic systems.