On-line learning algorithms for locally recurrent neural networks

被引:115
作者
Campolucci, P [1 ]
Uncini, A
Piazza, F
Rao, BD
机构
[1] Univ Ancona, Dipartimento Elettron & Automat, I-60131 Ancona, Italy
[2] Univ Calif San Diego, Dept Elect & Comp Engn, La Jolla, CA 92037 USA
来源
IEEE TRANSACTIONS ON NEURAL NETWORKS | 1999年 / 10卷 / 02期
关键词
IIR-MLP; locally recurrent neural networks; on-line learning; recursive backpropagation; system identification; time-delay neural networks;
D O I
10.1109/72.750549
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
This paper focuses on on-line learning procedures for locally recurrent neural networks with emphasis on multilayer perceptron (MLP) with infinite impulse response (IIR) synapses and its variations which include generalized output and activation feedback multilayer networks (MLN's). We propose a new gradient-based procedure called recursive backpropagation (RBP) whose on-line version, causal recursive backpropagation (CRBP), presents some advantages with respect to the other on-line training methods. The new CRBP algorithm includes as particular cases backpropagation (BP), temporal backpropagation (TBP), backpropagation for sequences (BPS), Back-Tsoi algorithm among others, thereby providing a unifying view on gradient calculation techniques for recurrent networks with local feedback. The only learning method that has been proposed for locally recurrent networks with no architectural restriction is the one by Back and Tsoi, The proposed algorithm has better stability and higher speed of convergence with respect to the Back-Tsoi algorithm, which is supported by the theoretical development and confirmed by simulations. The computational complexity of the CRBP is comparable with that of the Back-Tsoi algorithm, e,g,, less that a factor of 1.5 for usual architectures and parameter settings. The superior performance of the new algorithm, however, easily justifies this small increase in computational burden. In addition, the general paradigms of truncated BPTT and RTRL are applied to networks with local feedback and compared with the new CRBP method. The simulations show that CRBP exhibits similar performances and the detailed analysis of complexity reveals that CRBP is much simpler and easier to implement, e,g,, CRBP is local in space and in time while RTRL is not local in space.
引用
收藏
页码:253 / 271
页数:19
相关论文
共 63 条