GRADIENT DESCENT LEARNING ALGORITHM OVERVIEW - A GENERAL DYNAMICAL-SYSTEMS PERSPECTIVE

We give a unified treatment of gradient descent learning algorithms for neural networks using a general framework of dynamical systems, This general approach organizes and simplifies all the known algorithms and results which have been originally derived for different problems (fixed point/trajectory learning), for different model (discrete/continuous), for different architectures (forward/recurrent), and using different techniques (backpropagation, variational calculus, adjoint methods, etc.). The general approach can also be applied to derive new algorithms. We then briefly examine some of the complexity issues and limitations intrinsic to gradient descent learning. Throughout the paper, we focus on the problem of trajectory learning.