Stability of steepest descent with momentum for quadratic functions

被引：36

作者：

Torii, M ^{[1
]}

Hagan, MT

机构：

[1] Univ Delaware, Newark, DE 19711 USA

[2] Oklahoma State Univ, Stillwater, OK 74074 USA

来源：

IEEE TRANSACTIONS ON NEURAL NETWORKS | 2002年 / 13卷 / 03期

关键词：

convergence speed; gradient descent; momentum; stability;

D O I：

10.1109/TNN.2002.1000143

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

This paper analyzes the effect of momentum on steepest descent training for quadratic performance functions. We demonstrate that there always exists a momentum coefficient that will stabilize the steepest descent algorithm, regardless of the value of the learning rate. We also demonstrate how the value of the momentum coefficient changes the convergence properties of the algorithm.

引用

页码：752 / 756

页数：5

共 8 条

[1]

Brogan W. L., 1991, Modern Control Theory

[2]

Hagan MT., 1996, NEURAL NETWORK DESIG

[3]

HAGIWARA M, 1995, IEICE T INFORM SYS D, V78

[4]

PHANSALKAR VV, 1994, IEEE T NEURAL NETWOR, V5

[5] On the momentum term in gradient descent learning algorithms [J].

Qian, N .

NEURAL NETWORKS, 1999, 12 (01) :145-151

[6] LEARNING REPRESENTATIONS BY BACK-PROPAGATING ERRORS [J].

RUMELHART, DE ;

HINTON, GE ;

WILLIAMS, RJ .

NATURE, 1986, 323 (6088) :533-536

[7]

SAITO A, 1991, P ICANN91, P617

[8]

Werbos P. J., 1994, The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting, V1

← 1 →