The discovery of Ly alpha absorption lines at lower redshifts raises the possibility of a detailed study of the relation between absorbers and galaxies, and hence of determining a good model for the origin of the lines. In this paper, we set constraints on the possible fractions of the observed absorbers that could be associated with galaxies. We combine this result with other observations to set constraints on models for the absorbers. The minihalo model, where the absorbers are confined by the gravity of cold dark matter, predicts correctly the observed number of absorbers and the weak correlation between absorbers and galaxies on large scales, but fails to explain the close association of some absorbers with galaxies. Models where absorbers are produced by galaxy haloes can account for only about 20 per cent of the observed absorbers, unless the cross-sections of galaxies are huge and the cross-correlation on large scales is significantly underestimated. We propose that the observed absorbers are a combination of these two populations, with the absorbers closely associated with galaxies being about 20 per cent of the total. Such a model is consistent with observations of both the number of absorbers and the cross-correlation between absorbers and galaxies. The absorbers closely associated with galaxies may be pressure-confined by the ambient hot gas of galaxies or groups of galaxies.