Life history theory often assumes a positive relationship between offspring size and fitness, although the strength and form of this relationship is expected to vary with environmental conditions. In arthropods, surprisingly few studies have examined the influence of larval environment on the offspring size-fitness relation. In phytophagous insects, the few studies that have examined variation in larval host plants have found a negative correlation between host plant nutritional quality and the strength of selection favoring larger offspring size, suggesting that this pattern might be general. I present experimental evidence for such a relationship in a population of the moth Rothschildia lebeau feeding on its three primary host plant species. Unlike previous studies, I consider the effect of offspring size on growth and survival at two levels, both among families and among siblings within families. Neonate caterpillar mass had a significant effect on growth and survival. The effect on growth, however, was weak, resulted primarily from variation among families, and did not differ among host plant diets. The effect on survival was stronger and varied among host plant diets, among families, and within families on different host plants in a manner that was generally consistent with the hypothesized negative correlation between host plant nutritional quality and the strength of selection favoring larger offspring size. Overall, results suggest that the consequences of variation in offspring size for survival within and among families are host plant-dependent in this system.