A FORWARD METHOD FOR OPTIMAL STOCHASTIC NONLINEAR AND ADAPTIVE-CONTROL

A computational approach is taken to solve the optimal partially observed nonlinear stochastic control problem. The approach is to systematically solve the stochastic dynamic programming equations forward in time, using a nested stochastic approximation technique. Although computationally intensive, this provides a straightforward numerical solution for this class of problems and provides an alternative to the usual "curse of dimensionality" associated with solving the dynamic programming equations backwards in time. In particular, the "curse" is seen to take a new form, where the amount of computation depends on the amount of uncertainty in the problem and the length of the horizon. As a matter of more practical interest, it is shown that the cost degrades monotonically as the complexity of the algorithm is reduced. This provides a strategy for suboptimal control with clear performance/computation trade-offs. Due to a strong connection with the work of Bellman on policy iteration, the method is denoted as the iteration in policy space (IPS) algorithm. A numerical study focusing on a generic optimal stochastic adaptive control example is included to demonstrate the feasibility of the method.