GRADIENT CALCULATIONS FOR DYNAMIC RECURRENT NEURAL NETWORKS - A SURVEY

We survey learning algorithms for recurrent neural networks with hidden units and put the various techniques into a common framework, We discuss fixed point learning algorithms, namely recurrent backpropagation and deterministic Boltzmann machines, and nonfixed point algorithms, namely backpropagation through time, Elman's history cutoff, and Jordan's output feedback architecture, Forward propagation, an on-line technique that uses adjoint equations, and variations thereof, are also discussed, In many cases, the unified presentation leads to generalizations of various sorts, We discuss advantages and disadvantages of temporally continuous neural networks in contrast to clocked ones continue with some ''tricks of the trade'' for training, using, and simulating continuous time and recurrent neural networks, We present some simulations, and at the end, address issues of computational complexity and learning speed.