We discuss constraints on the assembly history of supermassive black holes from the observed remnant black holes in nearby galaxies and from the emission caused by accretion onto these black holes. We also summarize the results of a specific model for the evolution of galaxies and their central black holes which traces their hierarchical build-up in CDM-like cosmogonies. The model assumes (i) that black holes, ellipticals and starbursts form during major mergers of galaxies (ii) that the gas fraction in galaxies decreases with decreasing redshift (iii) that the optically bright phase of a QSO lasts for about 10(7) years. The model successfully reproduces the evolution of cold gas as traced by damped Ly alpha systems, the evolution of optically bright QSOs, the remnant black hole mass distribution and the host-galaxy luminosities of QSOs.