Misspecifying the likelihood for clustered binary data

The effect of misspecifying the parametric response model for a clustered binary outcome from a toxicological study on the assessment of dose effect is investigated. A marginal, random effects, and conditional model are contrasted, with the emphasis on likelihood based estimation. The methods are compared through asymptotic calculations, by means of small sample simulations, and on real developmental toxicity data. It is found that the beta-binomial and conditional models exhibit satisfactory behavior in terms of testing the null hypothesis of no dose effect. Whereas the conditional model has clear computational advantages, parameters in the beta-binomial model have a straightforward marginal interpretation. (C) 1998 Elsevier Science B.V. All rights reserved.