THE ROLE OF LIKELIHOOD AND ENTROPY IN INCOMPLETE-DATA PROBLEMS - APPLICATIONS TO ESTIMATING POINT-PROCESS INTENSITIES AND TOEPLITZ CONSTRAINED COVARIANCES

The role of maximum entropy is explored for maximum-likelihood estimation (MLE) problems in which the measured data are statistical observations and moment constraints on the observation function do not exist. It is shown that for given finite observations of a random process, the set of maximum entropy (maxent) densities are identical to the set generated via rules of conditional probability. By viewing the measurements as specifying the domain over which the maxent density is defined, rather than as a moment constraint on the observation function, the maxent density closest to the prior in the cross-entropy sense is just a conditional density. Because of this identity, maximum-likelihood parameter estimates can be obtained by solving a joint maximization (minimization) of the entropy function (Kullback-Liebler divergence). This reduces to finding the parameters which maximize the expected value of the log of the prior, where the expectation is taken with respect to the maxent density. The authors also derive a recursive algorithm for the generation of Toeplitz constrained maximum-likelihood estimators. At each iteration the algorithm evaluates conditional mean estimates of the lag products based on the previous estimate of the covariance, from which the updated Toeplitz covariance is generated.