Constructive backpropagation for recurrent networks

Choosing a network size is a difficult problem in neural network modelling. In many recent studies, constructive or destructive methods that add or delete connections, neurons or layers have been studied in order to solve this problem. In this work we consider the constructive approach, which is in many cases a very computationally efficient approach. In particular, we address the construction of recurrent networks by the use of constructive backpropagation. The benefits of the proposed scheme are firstly that fully recurrent networks with an arbitrary number of layers can be constructed efficiently. Secondly, after the network has been constructed we can continue the adaptation of the network weights as well as we can of its structure. This includes both addition and deletion of neurons/layers in a computationally efficient manner. Thus, the investigated method is very flexible compared to many previous methods. In addition, according to our time series prediction experiments, the proposed method is competitive in terms of modelling performance and training time compared to the well-known recurrent cascade-correlation method.