Evaluating kindergarten retention policy: A case study of causal inference for multilevel observational data

This article considers the policy of retaining low-achieving children in kindergarten rather than promoting them to first grade. Under the stable unit treatment value assumption (SUTVA) as articulated by Rubin, each child at risk of retention has two potential outcomes: Y(1) if retained and Y(0) if promoted. But SUTVA is questionable, because a child's potential outcomes will plausibly depend on which school that child attends and also on treatment assignments of other children. We develop a causal model that allows school assignment and peer treatments to affect potential outcomes. We impose an identifying assumption that peer effects can be summarized through a scalar function of the vector of treatment assignments in a school. Using a large, nationally representative sample, we then estimate (1) the effect of being retained in kindergarten rather than being promoted to the first grade in schools having a low retention rate, (2) the retention effect in schools having a high retention rate, and (3) the effect of being promoted in a low-retention school as compared to being promoted in a high-retention school. This third effect is not definable under SUTVA. We use multilevel propensity score stratification to approximate a two-stage experiment. At the first stage, intact schools are blocked on covariates and then, within blocks, randomly assigned to a policy of retaining comparatively more or fewer children in kindergarten. At the second stage, "at-risk" students within schools are blocked on covariates and then assigned at random to be retained. We find evidence that retainees learned less on average than did similar children who were promoted, a result found in both high-retention and low-retention schools. We do not detect a peer treatment effect on low-risk students.