The presence of a chemical potential completely changes the analytical structure of the QCD partition function. In particular, the eigenvalues of the Dirac operator are distributed over a finite area in the complex plane, whereas the zeros of the partition function in the complex mass plane remain on a curve. In this paper we study the effects of the fermion determinant at a nonzero chemical potential on the Dirac spectrum by means of the resolvent G(z) of the QCD Dirac operator. The resolvent is studied both in a one-dimensional U(1) model (Gibbs model) and in a random matrix model with the global symmetries of the QCD partition function. In both cases we find that, if the argument z of the resolvent is not equal to the mass m in the fermion determinant, the resolvent diverges in the thermodynamic limit. However, for z=m the resolvent in both models is well defined. In particular, the nature of the limit z-->m is illuminated in the Gibbs model. The phase structure of the random matrix model in the complex m and mu planes is investigated both by a saddle point approximation and via the distribution of Yang-Lee zeros. Both methods are in complete agreement and lead to a well-defined chiral condensate and quark number density.