Andrew Gelman, in a recent post of suggestions for psychologists in improving their research practice, advises greater use of within-subject comparisons. That is, to use an example I used a few months ago, rather than randomizing half the Sneetches to get stars on their bellies and half to stay star-free (and then testing the effect of the star on their frankfurter-roasting ability), we should try testing the effect of the star on the same Sneetch over time, comparing his frankfurter roasting as he goes on and off of the star-belly performance-enhancing drug.
In the education literature, this is often called a single-case design–
It tends to be particularly popular for studying kids with behavioral or emotional disabilities, where it’s assumed that once you stop whatever interventions you’re using to help a kid stop hitting the other 2nd graders or crying uncontrollably, he’ll be back at square one and go back to crying uncontrollably or hitting the other 2nd graders. But for most educational studies, there’s an assumption that these kinds of designs won’t work because every intervention might be the Little Thing that Makes a Big Difference– you can’t give the kid the Magic Bullet (the once-a-week hour-long rap session that will transform his whole life, the admissions offer to the perfect charter school, the math computer game that will make him finally understand factoring quadratic polynomials) and then take it back away- he’s already been saved.
Personally, I think this is wrong- all real impacts fade out, because human beings forget as well as learn, and because forgetting is part of normal growth and development of the human animal. If you think you’ve found the magic bullet whose effects grow and grow and grow, it’s because the ignorability assumption has been violated and you’re not really looking at comparable groups.
Given that we should expect fade-out, and because as Gelman says, “poisoning the well is the least of our worries,” there’s a lot to be said for the single-case design, both because it controls for the differences among people effectively (by using the same people) in a way that regular randomized controlled trials only can do under ideal conditions, and because it requires smaller sample sizes and shorter turnarounds that would counter some of the extreme cost and slow turnaround of well-done randomized trials.
But this would require expecting fade-out, rather than viewing it as something that went wrong, and recognizing that, counter to the ever-present hunger for the final solution to social inequality, even the best programs and places will only have partial, temporary, transient effects, and that “Nature’s first green is gold/Her hardest hue to hold/Her early leaf’s a flower/But only so an hour/Then leaf subsides to leaf/So Eden sank to grief, /So dawn goes down to day./Nothing gold can stay.”