Schools are full of cutoffs: how many credits you need to graduate, the test score you need to avoid summer school or to get out of ESL class, how many checks next to your name on the board before you get sent to the principal’s office.
There is an increasingly popular evaluation design called Regression Discontinuity that makes use of cutoffs. The idea is to compare the kids right above the cutoff with those right below the cutoff, using regression to control for their difference in score. While, for example, you would expect the kids who got sent to summer school because they flunked the end-of-year math test to do worse in next year’s math course than the kids who passed, if you compare the outcomes of the kids who just barely flunked with the kids who just barely passed, and control for their test score, you can get a good picture of whether summer school helped them. Because kids can’t control if they get one point under or one point over the cutoff score very well, this design (in theory) might neutralize some of the unobserved differences in who enters a program, without the sturm und drang of making schools randomly assign some kids to summer school and some kids to summer vacation who got the exact same score on the test.
It’s a good enough study design in its way, but what if the important policy isn’t the program you are testing, but the cutoff itself? For example, one of the big controversies in math education (as I understand it) has long been how many kids can benefit from a full year of Algebra in middle school , with some zealots claiming that everyone can take it just fine, and the City of San Francisco deciding recently that if some kids can’t take Algebra in middle school, nobody can, and then taking its toys and going home.
A valuable use of federal research money, I think, would be to proffer grants to a reasonably large group of districts to agree to randomly agree to a test score cutoff for taking Algebra in middle school, each district assigned to a single cutoff across a wideish but reasonable range. Call it “random cutoff regression discontinuity.” It might make some people mad (what education policy or study doesn’t?) but it would tell us something useful about what works well for whom.
Of course, it would require an explicit acknowledgement that the appropriate course for kids varies depending on ability– a fact that literally everyone in education agrees on implicitly (thus the ubiquitous use of cutoffs) but no one likes to say.