Roland Fryer, the Harvard economist, is well-known for research showing positive effects of some “No Excuses” charter schools, and somewhat more impressively, for pulling out several practices of these schools and then working with Houston to implement and evaluate these practices in some of their district-run schools, with quite promising results.
A recent paper from Fryer and his colleague William Dobbie that finds negative results from Texas charter schools as a whole therefore came as something of a surprise:
We estimate the impact of charter schools on early-life labor market outcomes using administrative data from Texas. We find that, at the mean, charter schools have no impact on test scores and a negative impact on earnings. No Excuses charter schools increase test scores and four-year college enrollment, but have a small and statistically insignificant impact on earnings, while other types of charter schools decrease test scores, four-year college enrollment, and earnings. Moving to school-level estimates, we find that charter schools that decrease test scores also tend to decrease earnings, while charter schools that increase test scores have no discernible impact on earnings. In contrast, high school graduation effects are predictive of earnings effects throughout the distribution of school quality. The paper concludes with a speculative discussion of what might explain our set of facts.
The study uses exclusively non-experimental data; Fryer and Dobbie note that even charter schools that have lotteries for entry do not hold onto the records from these lotteries for more than a couple years.
My guess is that charter schools generally have zero impacts on average; this is what more nationally representative studies seem to show (and is what you’d expect from the Iron Law.) Moreover, a similar study to Fryer and Dobbie’s of Florida charter schools , from Tim Sass of Georgia State and colleagues, published earlier this year, showed the opposite– that attendees of charter high schools had higher earnings as young adults:
This paper is the first to estimate charter schools’ effects on earnings in adulthood, alongside effects on educational attainment. Using data from Florida, we first confirm previous research (Booker et al., 2011) that students attending charter high schools are more likely to graduate from high school and enroll in college. We then examine two longer-term outcomes not previously studied in research on charter schools—college persistence and earnings. We find that students attending charter high schools are more likely to persist in college, and that in their mid-20s they experience higher earnings.
So…maybe Florida charter schools are good and Texas charter schools are bad? It’s certainly possible. Certainly, in general, charter schools in places where it’s harder to start a charter school seem to be better, though I don’t know if that’s particularly true of Florida versus Texas. The thing about Dobbie and Fryer’s results that is puzzling, however, is that short-term test score results were approximately zero, but the negative impacts on earnings were quite large, and the impacts on employment were also approximately zero.
So, I’ll introduce Toad’s Corollary to the Iron Law:
If long-term impacts of a social intervention are larger than short-term impacts, in either the negative or positive direction, you’re probably missing something.
Okay, not so profound. But the point is that we should expect fade-out. People are changing all the time, and kids are really changing all the time, mostly in biologically determined ways and mostly having little to do with which school they attend. If a social intervention seems to do nothing for the first couple years and then totally screw up the kids several years down the line (when they haven’t had any interaction with the program in years)– maybe the kids were different to begin with in ways we didn’t notice.
So what could we (or Dobbie and Fryer in this case) be missing? If you look at the table above, you can see they did a whole bunch of robustness checks and different specifications, all of which appear to show these large negative impacts on earnings. At the very least, they appear to be controlling well for the influence of obvious factors like race, free-and-reduce-price-lunch status, and prior test scores. What else is likely to differ systematically between charter school kids and non-charter school kids, particularly in non-“No Excuses” charter schools, in a way that could change future earnings for individual young adults?
My guess is religiosity. A frequent criticism of charter schools is that they enable public school funds to be used to further de facto religious education, and there is, in the Washington Post’s words, “a growing body of literature about ‘religious’ and ‘faith-based’ charter schools,” along with frequent claims of religious practices being mixed into the school day. States with more religious conservatives appear to have policies that make it easier to start charter schools to begin with. The point is that charter schools do not need to be explicitly religious, but if they are attracting more religious families to enroll, this will have implications for their attendees’ outcomes down the line.
I did a little poking around with General Social Survey data to see how approval of school prayer and attending religious services more than once a month were correlated or not with earnings at age 23 to 28 (Fryer took from aged 24 to 26, but the GSS samples are fairly small); the correlation is stronger for attendance than for approving of school prayer, but it does suggest that depending on the racial balance in Texas charter schools, religiosity could influence later earnings a fair amount.
Because of the way GSS encodes earnings, about a third of respondents this age are just placed into “over $25,000,” which isn’t terribly helpful; I encoded these as 32,500, which produced average earnings across the whole sample fairly similar to Fryer’s sample.
I looked at female respondents because my guess was that having kids was the most likely mechanism for religiosity to influence individual earnings among young adults:
Indeed, opinions about abortion, another indicator of religiosity, are also associated with incomes, with strong pro-choice views more prevalent among young women earning $25,000 or more, and pro-life views more common among every group with lower incomes:
Why might the association of charter attendance with religiosity show up in Texas’s results but not Florida’s? Because Texas is a fair amount more religious in general than Florida:
It’s easy to think of reasons this particular hypothesis might be wrong: maybe the Texas charter schools really were godawful and screwed up the kids for good. Maybe there were unobserved difference between the kids going to non-“No Excuses” charter schools and the kids in regular district schools, but those differences had nothing to do with religiosity (or religious beliefs in young adulthood are a result rather than a cause of lower incomes.)
It’s also easy to come up with examples that violate Toad’s Corollary in general: if you get a 17-year-old kid drunk or give him a concussion one Friday night before he takes the SAT test, it’s conceivable that he’d be fine a week later but his earnings might suffer ten years down the line.
But at a first pass: if long-term impacts appear bigger than short-term impacts, we should take them with an extra helping of salt.