Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

被引：67

作者：

Bhaya, A ^{[1
]}

Kaszkurewicz, E ^{[1
]}

机构：

[1] Univ Fed Rio de Janeiro, COPPE, PEE, Dept Elect Engn, BR-21945970 Rio de Janeiro, RJ, Brazil

来源：

NEURAL NETWORKS | 2004年 / 17卷 / 01期

关键词：

backpropagation; steepest descent; momentum; conjugate gradient algorithm; convergence; continuous optimization; bilinear system; control Liapunov function;

D O I：

10.1016/S0893-6080(03)00170-9

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

It is pointed out that the so called momentum method, much used in the neural network literature as an acceleration of the backpropagation method, is a stationary version of the conjugate gradient method. Connections with the continuous optimization method known as heavy ball with friction are also made. In both cases, adaptive (dynamic) choices of the so called learning rate and momentum parameters are obtained using a control Liapunov function analysis of the system. (C) 2003 Elsevier Ltd. All rights reserved.

引用

页码：65 / 71

页数：7

共 23 条

[1]

ALBER YI, 1971, DIFF EQUAT, V7, P1461

[2] The heavy ball with friction method, I. The continuous dynamical system: Global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipative dynamical system [J].

Attouch, H ;

Goudou, X ;

Redont, P .

COMMUNICATIONS IN CONTEMPORARY MATHEMATICS, 2000, 2 (01) :1-34

[3]

BHAYA A, 2002, NACADBK0102

[4] ON THE CONTINUOUS REALIZATION OF ITERATIVE PROCESSES [J].

CHU, MT .

SIAM REVIEW, 1988, 30 (03) :375-387

[5] MATRIX DIFFERENTIAL-EQUATIONS - A CONTINUOUS REALIZATION PROCESS FOR LINEAR ALGEBRA PROBLEMS [J].

CHU, MT .

NONLINEAR ANALYSIS-THEORY METHODS & APPLICATIONS, 1992, 18 (12) :1125-1146

[6] FEEDBACK STABILIZATION OF A 2ND-ORDER SYSTEM - A NONMODAL APPROACH [J].

DATTA, BN ;

RINCON, F .

LINEAR ALGEBRA AND ITS APPLICATIONS, 1993, 188 :135-161

[7]

Goodwin GC, 2001, CONTROL SYSTEM DESIGN, pXXIII

[8]

Greenbaum A., 1997, Iterative methods for solving linear systems

[9]

HAGIWARA M, 1995, IEICE T INF SYST, VE78D, P1080

[10]

Haykin S., 1999, NEURAL NETWORK COMPR

← 1 2 3 →