The Iron Law

The Iron Law of Evaluation (Rossi, 1987) is that the expected value of any net impact assessment of any large scale social program is zero.

Of course, if the expected value is zero, that could just mean that there are as many negatives as positives. This is more likely to occur, of course, in preregistered randomized trials, where the researchers have fewer degrees of freedom to tiptoe through the tulips in the garden of forking paths. (This is the “stainless steel law” of policy evaluation, Ibid.) It also is more likely to happen when researchers’ incentives are to appear (or to be) unbiased, rather than to support a particular intervention. Carol Dweck is never, ever going to discover that growth mindset is actively harmful, no matter what trial she runs or what outcome she examines.  But it does happen.

For example, the federal government’s largest-scale randomized trial of labor programs for assisting low-income people in the 2000s, the Employment, Retention, and Advancement program, found statistically significant positive effects for three programs, statistically significant negative effects for two programs, and null effects for another seven. The evaluation of Building Strong Families, the Bush Administration’s >800 million dollar initiative to encourage marriage among low-income parents, found null effects in six programs, positive effects in one in Oklahoma, and strong negative effects in another in Baltimore, as well as statistically significant negative impacts on two outcomes when the eight programs were aggregated together.

Why does this happen? The simple answer is that in a largely-rich, largely-free country, with many existing (if confusing) private and public supports for low-income people, it’s just as easy to screw things up than to make things better, no matter how much you spend.

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s