To explore the relationship between local electromagnetic field enhancement and the large SERS (surface enhanced Raman scattering) enhancement that enables the observation of single molecule Raman spectra, we measure both resonant Rayleigh scattering spectra and rhodamine 6G Raman spectra from single Ag particles. Our apparatus combines the techniques of dark-field optical microscopy for resonant Rayleigh measurements, and grazing incidence Raman spectroscopy. The Rayleigh spectra show that the citrate-reduced AE colloid is extremely heterogeneous. Only the larger particles, in part created by salt induced aggregation, show a large SERS effect. In agreement with the work of Nie and Emery, we find that a few nanocrystals show huge single molecule R6G SERS intensities. While all SERS active particles have some resonant Rayleigh scattering at the 514.5 nm laser wavelength, there is no correlation between the resonant Rayleigh spectra and the SERS intensity. We discuss a model in which huge SERS intensities result from single chemisorbed molecules interacting with ballistic electrons in optically excited large Ag particles. This model is a natural consequence of the standard local electromagnetic field model for SERS and the high surface sensitivity of plasmon dephasing in the noble metals.