IMPROVING GENERALIZATION PERFORMANCE USING DOUBLE BACKPROPAGATION

被引:158
作者
DRUCKER, H
LECUN, Y
机构
[1] MONMOUTH COLL, LONG BRANCH, NJ 07764 USA
[2] AT&T BELL LABS, HOLMDEL, NJ 07733 USA
来源
IEEE TRANSACTIONS ON NEURAL NETWORKS | 1992年 / 3卷 / 06期
关键词
12;
D O I
10.1109/72.165600
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
In order to generalize from a training set to a test set, it is desirable that small changes in the input space of a pattern do not change the output components. This can be done by including variations of the input space as part of the training set-but this is computationally very expensive. Another method is to force this behavior as part of the training algorithm. This is done in double backpropagation by forming an energy function that is the sum of the normal energy term found in backpropagation and an additional term that is a function of the Jacobian. Significant improvement is shown with different architectures and different test sets, especially with architectures that had previously been shown to have very good performance when trained using backpropagation. It also will be shown that double backpropagation, as compared to backpropagation, creates weights that are smaller thereby causing the output of the neurons to spend more time in the linear region.
引用
收藏
页码:991 / 997
页数:7
相关论文
共 12 条
[1]  
[Anonymous], 1990, ADV NEURAL INF PROCE
[2]  
Gallant S. I., 1986, Eighth International Conference on Pattern Recognition. Proceedings (Cat. No.86CH2342-4), P849
[3]  
Hinton G.E., 1986, P 8 ANN C COGN SCI S, V1, P12
[4]  
KRAMER AH, 1989, ADV NEURAL INFORMATI, V1
[5]   Backpropagation Applied to Handwritten Zip Code Recognition [J].
LeCun, Y. ;
Boser, B. ;
Denker, J. S. ;
Henderson, D. ;
Howard, R. E. ;
Hubbard, W. ;
Jackel, L. D. .
NEURAL COMPUTATION, 1989, 1 (04) :541-551
[6]  
LeCun Y., 1989, P ADV NEURAL INFORM
[7]  
Lecun Y., CONNECTIONISM PERSPE
[8]   LEARNING IN FEEDFORWARD LAYERED NETWORKS - THE TILING ALGORITHM [J].
MEZARD, M ;
NADAL, JP .
JOURNAL OF PHYSICS A-MATHEMATICAL AND GENERAL, 1989, 22 (12) :2191-2203
[9]  
Rumelhart David E., 1987, LEARNING INTERNAL RE, P318
[10]  
SCALETTAR R, 1988, CONNECTIONIST MODELS, P309