The extent to which the effect of risk factors on cognitive ageing is dependent on APOE epsilon 4 remains unclear. The objective of this study is to examine whether APOE epsilon 4 allele modifies the association between health behaviors and cognition in late midlife. Data are drawn from 5447 participants of the Whitehall II study, health behaviors were assessed in 1997-1999 (mean age = 55.6, Standard Deviation (SD) = 6.0) and APOE genotype and cognitive function in 2002-2004 (mean age = 60.9, SD = 5.9). Among APOE epsilon 4 non-carriers, current smokers had lower scores on memory (difference in T-score = -2.49, 95% CI: -3.37, -1.60), reasoning (-2.88, 95% CI: -3.74, -2.01), phonemic (-2.66, 95% CI: -3.56, -1.76) and semantic (-2.38, 95% CI: -3.28, -1.47) fluency compared to never smokers. In APOE epsilon 4 carriers, difference between current and never smokers was seen only for reasoning (-1.92, 95% CI: -3.31, -0.51). Interaction terms supported differential effects of smoking as a function of APOE epsilon 4 status for memory (p = 0.01), and phonemic (p = 0.008) and semantic fluency (p = 0.02). Cognitive scores were lower among non-drinkers compared to moderate drinkers, among the sedentary participants and those who ate fruits and vegetable less than 2 times per day irrespective of APOE epsilon 4 status. This study suggests that the APOE epsilon 4 allele modifies the association of smoking but not that of other health behaviors - alcohol consumption, physical activity, fruit and vegetable consumption - with cognitive function in late midlife.