Toad’s Corollary

Roland Fryer, the Harvard economist, is well-known for research showing positive effects of some “No Excuses” charter schools, and somewhat more impressively, for pulling out several practices of these schools and then working with Houston to implement and evaluate these practices in some of their district-run schools, with quite promising results.

A recent paper from Fryer and his colleague William Dobbie that finds negative results from Texas charter schools as a whole therefore came as something of a surprise:

We estimate the impact of charter schools on early-life labor market outcomes using administrative data from Texas. We find that, at the mean, charter schools have no impact on test scores and a negative impact on earnings. No Excuses charter schools increase test scores and four-year college enrollment, but have a small and statistically insignificant impact on earnings, while other types of charter schools decrease test scores, four-year college enrollment, and earnings. Moving to school-level estimates, we find that charter schools that decrease test scores also tend to decrease earnings, while charter schools that increase test scores have no discernible impact on earnings. In contrast, high school graduation effects are predictive of earnings effects throughout the distribution of school quality. The paper concludes with a speculative discussion of what might explain our set of facts.

The study uses exclusively non-experimental data; Fryer and Dobbie note that even charter schools that have lotteries for entry do not hold onto the records from these lotteries for more than a couple years.

My guess is that charter schools generally have zero impacts on average; this is what more nationally representative studies seem to show (and is what you’d expect from the Iron Law.) Moreover, a similar study to Fryer and Dobbie’s of Florida charter schools , from Tim Sass of Georgia State and colleagues, published earlier this year, showed the opposite– that attendees of charter high schools had higher earnings as young adults:

 This paper is the first to estimate charter schools’ effects on earnings in adulthood, alongside effects on educational attainment. Using data from Florida, we first confirm previous research (Booker et al., 2011) that students attending charter high schools are more likely to graduate from high school and enroll in college. We then examine two longer-term outcomes not previously studied in research on charter schools—college persistence and earnings. We find that students attending charter high schools are more likely to persist in college, and that in their mid-20s they experience higher earnings.

So…maybe Florida charter schools are good and Texas charter schools are bad? It’s certainly possible. Certainly, in general, charter schools in places where it’s harder to start a charter school seem to be better, though I don’t know if that’s particularly true of Florida versus Texas. The thing about Dobbie and Fryer’s results that is puzzling, however, is that short-term test score results were approximately zero, but the negative impacts on earnings were quite large, and the impacts on employment were also approximately zero.

CHarterEarnings

So, I’ll introduce Toad’s Corollary to the Iron Law:

If long-term impacts of a social intervention are larger than short-term impacts, in either the negative or positive direction, you’re probably missing something.

Okay, not so profound. But the point is that we should expect fade-outPeople are changing all the time, and kids are really changing all the timemostly in biologically determined ways and mostly having little to do with which school they attend. If a social intervention seems to do nothing for the first couple years and then totally screw up the kids several years down the line (when they haven’t had any interaction with the program in years)– maybe the kids were different to begin with in ways we didn’t notice.

So what could we (or Dobbie and Fryer in this case) be missing? If you look at the table above, you can see they did a whole bunch of robustness checks and different specifications, all of which appear to show these large negative impacts on earnings. At the very least, they appear to be controlling well for the influence of obvious factors like race, free-and-reduce-price-lunch status, and prior test scores. What else is likely to differ systematically between charter school kids and non-charter school kids, particularly in non-“No Excuses” charter schools, in a way that could change future earnings for individual young adults?

My guess is religiosity. A frequent criticism of charter schools is that they enable public school funds to be used to further de facto religious education, and there is, in the Washington Post’s words, “a growing body of literature about ‘religious’ and ‘faith-based’ charter schools,” along with frequent claims of religious practices being mixed into the school day. States with more religious conservatives appear to have policies that make it easier to start charter schools to begin with.  The point is that charter schools do not need to be explicitly religious, but if they are attracting more religious families to enroll, this will have implications for their attendees’ outcomes down the line.

I did a little poking around with General Social Survey data to see how approval of school prayer and attending religious services more than once a month  were correlated or not with earnings at age 23 to 28 (Fryer took from aged 24 to 26, but the GSS samples are fairly small); the correlation is stronger for attendance than for approving of school prayer, but it does suggest that depending on the racial balance in Texas charter schools, religiosity could influence later earnings a fair amount.

Because of the way GSS encodes earnings, about a third of respondents this age are just placed into “over $25,000,” which isn’t terribly helpful; I encoded these as 32,500, which produced average earnings across the whole sample fairly similar to Fryer’s sample.

prayer

attendance

I looked at female respondents because my guess was that having kids was the most likely mechanism for religiosity to influence individual earnings among young adults:

kidsornot

Indeed, opinions about abortion, another indicator of religiosity, are also associated with incomes, with strong pro-choice views more prevalent among young women earning $25,000 or more, and pro-life views more common among every group with lower incomes:

abortion

Why might the association of charter attendance with religiosity show up in Texas’s results but not Florida’s? Because Texas is a fair amount more religious in general than Florida:

evcrebyusstate

It’s easy to think of reasons this particular hypothesis might be wrong: maybe the Texas charter schools really were godawful and screwed up the kids for good. Maybe there were unobserved difference between the kids going to non-“No Excuses” charter schools and the kids in regular district schools, but those differences had nothing to do with religiosity (or religious beliefs in young adulthood are a result rather than a cause of lower incomes.)

It’s also easy to come up with examples that violate Toad’s Corollary in general: if you get a 17-year-old kid drunk or give him a concussion one Friday night before he takes the SAT test, it’s conceivable that he’d be fine a week later but his earnings might suffer ten years down the line.

But at a first pass: if long-term impacts appear bigger than short-term impacts, we should take them with an extra helping of salt.

7 thoughts on “Toad’s Corollary

  1. I’m not sure the implication of this is particularly clear:

    “Why might the association of charter attendance with religiosity show up in Texas’s results but not Florida’s? Because Texas is a fair amount more religious in general than Florida”

    Are you suggesting that Texas is also more uniformly religious, so charter schools aren’t as likely to pick out the most religious families?

    Like

    1. Let’s say being evangelical makes you 40 percent more likely to go to a charter school, and also reduces your earnings in your mid-20s conditioning on prior test scores, etc; a state with more religious students will show a larger earnings penalty.

      Like

  2. Not sure this is the best post to make the following comment, but here it is:

    I became increasingly pessimistic about a vast array of policy proposals in the past few years – and I really think that’s the rational response to finding out that, well, almost nothing works. Almost every government has some kind of job training program – yet those programs rarely work. Almost everyone agrees that good education is essential to reducing inequality of opportunity and creating better societies… but education doesn’t necessarily “work”. IQ is more important to life outcomes, more heritable, and (much) less actionable than people would like. I’m sure you’re at least as pessimistic as I am about all things shared environment. But we seem to reach different conclusions about what to make of that.
    My thinking goes something like this:
    1. Price systems and market economies are good ideas, in the sense that they create incentive structures that lead to long term prosperity.
    2. The problem about wages is the diminishing marginal utility of income, which can justify a certain level of redistribution (not total redistribution, of course, see #1). Even without thinking in those terms, some people can’t afford food and shelter with their market income only, and we don’t want those people to starve.
    3. In response to that, we can provide them with either money or stuff. Money is generally preferable, for two basic reasons: people know more about their own preferences than the state does, and businesses are generally more efficient at producing stuff than governments are. Exceptions may exist in case of, say, huge economies of scale or market failures.
    4. The fact that most government programs are clearly less beneficial than just giving people (the equivalent) money, and that some of those programs may not be beneficial at all, makes me much more inclined to favor cash transfers. Granted, cash transfers are no panacea, but they seem obviously superior to doing nothing to help people, and obviously superior to most things governments do.
    This is basically how people like Chris Blattman think too (see for instance http://chrisblattman.com/2013/05/23/dear-governments-want-to-help-the-poor-and-transform-your-economy-give-people-cash/).
    However, you don’t seem to think cash transfers are good – you’ve mentioned them here (https://spottedtoad.wordpress.com/2016/04/15/west-virginia-or-the-bad-news-takes-a-long-time-to-arrive/) and here (https://spottedtoad.wordpress.com/2016/02/01/poverty-is-not-easy-to-solve/). Do you disagree with any of my four points? If so, could you please elaborate on that?

    Like

    1. Yeah, this is an important topic, maybe the important topic of the 21st century as technology displaces more labor. Some kind of universal transfer is probably inevitable, but I think making that compatible with functional civil society, individual political rights in a more than a client/benefactor sense, and stable family structure isn’t at all a foregone conclusion. I’ll try to give a more complete answer as a post.

      Like

      1. These don’t exactly answer your question but are somewhat related-

        https://spottedtoad.wordpress.com/2016/01/26/poverty-and-cash-transfers/

        https://spottedtoad.wordpress.com/2016/02/17/the-race-to-the-swift-and-the-merits-of-meritocracy/

        https://spottedtoad.wordpress.com/2016/01/24/the-beloved-community-and-the-bourgeois-society/

        https://spottedtoad.wordpress.com/2016/03/08/trenton-makes-the-world-takes/

        https://spottedtoad.wordpress.com/2016/03/14/school-lunched/

        FWIW, I think our existing (in-kind mainly) means-tested system is a reasonable compromise, given the US population and political system. SNAP has its problems, but it means fewer hungry kids. I’d like the US system to disfavor marriage less and our health care system to be more efficient, but everyone wants a pony.

        Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s