Some species and individuals are able to learn cognitive skills more flexibly than others. Learning experiences and cortical function are known to contribute to such differences, but the specific factors that determine an organism's intellectual capacities remain unclear. Here, an integrative framework is presented suggesting that variability in cognitive plasticity reflects neural constraints on the precision and extent of an organism's stimulus representations. Specifically, it is hypothesized that cognitive plasticity depends on the number and diversity of cortical modules that an organism has available as well as the brain's capacity to flexibly reconfigure and customize networks of these modules. The author relates this framework to past proposals on the neural mechanisms of intelligence, including (a) the relationship between brain size and intellectual capacity; (b) the role of prefrontal cortex in cognitive control and the maintenance of stimulus representations; and (c) the impact of neural plasticity and efficiency on the acquisition and performance of cognitive skills. The proposed framework provides a unified account of variability in cognitive plasticity as a function of species, age, and individual, and it makes specific predictions about how manipulations of cortical structure and function will impact intellectual capacity.