We study the van der Waals interaction V(2 gamma)(AB)(R) arising from two-photon exchange between neutral spinless systems A and B. By using the analytic properties of the two-photon contribution to the scattering amplitude for A + B -> A + B and of the full amplitudes for gamma + A -> gamma + A and gamma + B -> gamma + B, we show that it is possible to express V(2 gamma)(AB)(R) entirely in terms of measurable quantities, the elastic scattering amplitudes for photons of various frequencies omega. This approach includes relativistic corrections, higher multipoles, and retardation effects from the outset and thus avoids any nu/c expansion or any direct reference to the detailed structure of the systems involved. We obtain a generalized form of the Casimir-Polder potential, which Includes both electric and magnetic effects, and, correspondingly, a generalized asympotic form V(2 gamma)(AB)(gamma) similar to -D/R(eta), where D = [23(alpha(A)(E)alpha(B)(E) + alpha(A)(M)alpha(B)(M)) - 7(alpha(A)(E)alpha(B)(M) + alpha(A)(M)alpha(B)(E))]/4 pi and the alpha's denote static polarizabilities. In addition, we show that the potential may be written as a single integral over omega, involving products of the dynamical polarizabilities alpha(X)(omega) evaluated at real frequencies, in contrast to the familiar integral over imaginary frequencies; for the case of interacting atoms, the domain of applicability of the various formulas Is clarified, and the problem of evaluating V(2 gamma)(AB)(R) from present experimental Information is discussed. Some simple interpolation formulas are presented, which may accurately describe V(2 gamma)(AB)(R) in terms of a few constants.