We perform ab initio calculations within the Landauer formalism to study the influence of doping on the conductance of surface-passivated silicon nanowires. It is shown that impurities located in the core of the wire induce a strong resonant backscattering at the impurity bound state energies. Surface dangling bond defects have hardly any direct effect on conductance, but they strongly trap both p- and n-type impurities, as evidenced in the case of H-passivated wires and Si/SiO2 interfaces. Upon surface trapping, impurities become transparent to transport, as they are electrically inactive and do not induce any resonant backscattering.