The structure of Ag-N clusters (N=1-4, 6, 8, 10), both in the gas phase and grown on the MgO(100) surface containing F-s-defects, has been investigated by a density functional basin-hopping (DF-BH) approach. In analogy with what observed in the case of gold clusters, it is found that the presence of the defect implies a double frustration and a cylindrical invariance of the metal-surface interaction, causing small Ag clusters growing around the F-s defect to be highly fluxional. Nevertheless, two different structural crossovers are found to be induced by the metal-defect interaction for the adsorbed clusters such that: 1) planar structures prevail for N <= 4 (as in the gas phase); 2) noncrystalline (fivefold symmetric) structures, which are the lowest energy ones in the gas phase for medium sized Ag-N clusters (N >= 7), prevail for N=6 and N=8; 3) distorted face-centered cubic (fcc) structures grown pseudomorphically on the defected surface prevail for N=10. The transition from fivefold to fee motifs is rationalized in terms of the double-frustration effect, which increases the bond strain of the noncrystalline structures. Detrapping energies from the defect were also calculated; the lowest energy pathway corresponds to the detachment of a dimer.