One of the reasons economists keep studying charter schools is that, because parents have to choose to sign up for them, there’s the chance the school will use a lottery to see which kids get in. The lottery itself, with tear-streaked or jubilant kids’ faces as a result, is a frequent image in documentaries hailing charter schools’ success. Economists love lotteries because, if well-conducted, they are the social science equivalent of a rainbow-colored unicorn: a randomized controlled trial that doesn’t cost much and that the program you are studying already wants to do.
Most randomized controlled trials of social programs are very, very expensive- hundreds of thousands to millions of dollars if you want to include enough participants to measure fairly small effects, and in education, even the biggest effects tend to be pretty small. Just tracking down low-income people to agree to be part of the study and to do an in-person interview with them at the beginning of a study or partway in can be a couple hundred bucks per interview; getting young kids to take a test or some kind of language assessment if they aren’t old enough to take standardized tests can cost hundreds more. This is mostly about getting decently representative samples and not losing all the people who sign up for your program and then move away, or go to jail, or change their name, or just decide not to pick up the phone ever again. One reason it drove me crazy that Michael LaCour’s advisers and would-be employer didn’t spot his data forgery well in advance was that the kind of multiple-wave randomized trial with lots of respondents he claimed to have conducted- even if the baseline data was collected by a group of volunteers- would have cost hundreds of thousands of dollars to get anywhere like a 70% response rate. LaCour claimed to have gotten responses from over 95% of his original sample using some entirely fictitious grants he won as a grad student and the promise of a chance to win a free IPad to people who answered the survey.
Once kids are old enough to take standardized tests, the costs for education studies can drop back down, since depending on the school’s policies, kids and their parents might not have to consent to the study at all (particularly if the test score data is used anonymously.) And the school is administering- has to administer- these tests anyways, so the costs of data collection have gone way down. Education Realist has often wondered aloud why Elementary and Middle School test scores are so often the buttress of evidence for charter school success, but it’s not just that grade 4-8 tests are easier and so the scores are easier to show progress in; it’s also that those are grades where the kids both have baseline test scores (from the year before) and are usually, thanks to No Child Left Behind, all taking the same standardized test at the end of the year. So the study has just gone from something only a federal Department can pay for to something a district or a few academics can make happen.
There’s still one big problem, even for studying kids in tested grades. Unless the kids are randomly assigned to the school or program you want to study, anyone looking at the study can blame unobserved factors like motivation or involved parents for whatever effects the study claims to show. But no school will agree to randomly assign students unless they are already at capacity, because otherwise they’ll be leaving empty seats (and losing funding attached to each kid) for no good reason.
The lottery solves all of these problems for the person doing the study. The school is already at capacity, which is why it has to use something like a lottery to auction off seats fairly in any case. The kids and their parents are already choosing to go to the school, so they’ll be more than willing to sign one more piece of paper to be part of the study. The original high-intensity charter schools were often grade 5-8 schools- exactly the grades that the researchers would want in terms of standardized test scores.
There’s just one problem here: the same reason that the school is easy to study is also the reason it is unlikely to be representative of other schools. This is already the school that has a line around the block of parents who want to go there, that’s why they’re doing the lottery. In spite of my numerous objections to randomized trials as an alpha and omega of learning about the world, there is a strong argument that a well-conducted lottery can give you a hint of a program’s “true causal effect,” particularly if you get those response rates high enough. But this same argument for the internal validity of the study for the kids who participate in it is also an equally strong argument for why the study doesn’t tell you much about the world of charter schools in general, which are the places that don’t have the intense parent demand and well-functioning administrators to pull off a lottery and have Harvard and MIT economists come study it. The Iron Law is still the Iron Law– if you could study perfectly every intervention going on simultaneously, the average impact would almost certainly be something like zero– but even the most honest and scrupulous of researchers can continuously report much more positive effects, if the same characteristics that makes something a reputable and affordable study also make it much more likely to find positive effects.
This isn’t just about schools- similar issues came up in studies of the Obama Administration’s energy efficiency programs; the first programs to be studied were also the ones that had their act together enough to show measurable effects.