VISUAL TRACKING OF A MOVING TARGET BY A CAMERA MOUNTED ON A ROBOT - A COMBINATION OF CONTROL AND VISION

In this paper, we present algorithms for robotic (eye-in-hand configuration) real-time visual tracking of arbitrary 3-D objects traveling at unknown velocities in a 2-D space (depth is given as known). We formulate the problem of visual tracking as a problem of combining control with computer vision. We present a mathematical formulation of a control problem that includes the sensory information of a novel and important feedback sensor (vision sensor). This formulation represents everything with respect to the camera frame and not with respect to the world frame. Due to this fact, we have the ability to quickly and accurately control the camera. We propose using the sum-of-squared differences (SSD) optical flow for the computation of the vector of discrete displacements each instant of time. These displacements can be fed either directly to a PI controller or to a pole assignment controller or to a discrete steady-state Kalman filter. In the latter case, the Kalman filter calculates the estimated values of the system's states and the exogenous disturbances, and a discrete LQG controller computes the desired motion of the robotic system. The outputs of the controllers are sent to a Cartesian robotic controller that drives the robot. The performance of the proposed algorithms has been tested on the CMU Direct-Drive (DD) Arm II, and the results are presented in this paper.