Genetic variation was shown earlier to be reduced in smaller populations of the narrow endemic putatively self-incompatible Cochlearia bavarica. To test whether this negatively affects plant fitness by reduced availability of compatible mates and by inbreeding depression, we studied effects of population size and pollination treatments on cross-compatibility and offspring fitness in 16 isolated populations of this plant. After open pollination, compatibility of crosses (i.e., whether at least one fruit developed per marked flower), fruit set of compatible crosses, and cumulative fitness (number of plants per maternal ovule) after 14 months in a common garden were lower for plants from smaller populations. Throughout the study, cumulative fitness was lower after hand pollination with pollen of one donor than after open pollination (finally 73.4% lower), suggesting that several pollen donors or single pollen donors of higher quality are involved in open pollination. Moreover, cumulative fitness was lower after hand selfing than after hand outcrossing (finally 69.4% lower), indicating both inbreeding depression and reduced compatibility after selfing. High self-compatibility (40.6%), dry stigmas, and differences in the compatibility of 11 of 33 experimental reciprocal crosses between plant pairs confirmed that C. bavarica has a sporophytic self-incompatibility system, as is common in the Brassicaceae. Our study demonstrates, that plants in smaller populations of species with a sporophytic self-incompatibility system can experience twofold fitness reductions associated with reduced genetic variability, i.e., twofold genetic Allee effects: via reduced cross-compatibility and via reduced offspring fitness.