It is often alleged that results from randomized evaluations like those of the Head Start Impact Study, which showed no pattern of positive effects at all by the 1st grade follow-up, demonstrate that the program they are evaluating is of no benefit, at least along the dimensions of child well-being or cognitive development the study examines.
This was, for example, the argument made by the Heritage Foundation in response to the results:
Recently released results from the Head Start Impact Study indicate that the benefits of participating in Head Start almost completely disappear by first grade. While other studies have previously assessed Head Start’s effectiveness, this is the only study that used a rigorous experimental design. Given this strongly negative evaluation, Congress should reconsider spending more than $9 billion per year on a program that produces few positive lasting effects. Furthermore, instead of creating yet another new federal preschool program at a cost of $8 billion, Congress and the Obama Administration should focus on terminating, consolidating, and reforming existing preschool and child care programs to better serve children’s needs and to improve efficiency for taxpayers.
This argument is incorrect. Randomized evaluations like the Impact Study of Head Start are informative about the marginal impact of offering a place in a program to a family. In the case of new or proposed programs, since a substantial portion of those participating in the program may be also participating in the study, and in any case the number of program participants is smaller than the number making use of alternatives, the marginal impact and the average impact is likely to be quite similar. The Impact Study of Head Start is suggestive that expansions in the number of families that Head Start serves or expansions to similar public pre-K programs for less disadvantaged children, will not produce particular benefits to child outcomes.
However, this does not, as the Heritage report suggests, mean that getting rid of Head Start- or other existing programs that serve a large percentage of potential recipients- would not cause any ill effects. Consider a crude diagram of distribution of low-income 4 year olds and how they are currently served:
The fact that a large percentage of the total percentage of low-income children are currently being served by Head Start indicates that the various non-Head Start options would have to expand their enrollment capacity substantially were Head Start to disappear– and it is uncertain that they would be able to maintain their current relative effectiveness were they to do so. This is particularly true if those served through Head Start are on average higher need than those successfully randomly assigned (about 15% of the 3-year-old control group and 20% of the 4-year-old control group managed to enroll in Head Start anyway, see Exhibit 2.4) or if public education programs produce positive spillover effects through peer or parent networks that benefit children who are not enrolled.
This may not be true at all. Any given public program may be of little use, no matter how popular or engrained in current patterns of behavior. But randomized impact evaluations are in general not as informative about these kinds of effects, or the value (or lack thereof) of programs that serve a large portion of potential recipients.