A researcher or group of researchers develops an intervention to test some hypothesis about what helps students learn. To be of academic interest, it usually can’t be something boring or obvious like “hire a bunch of tutors” or “make class sizes smaller” or “throw out all the kids that throw paper balls at each other,” because those tend to be really expensive (or illegal, in the paper balls case) and school districts already have made their classes as small as they can afford. Instead, the intervention to be tested tends to be something like Carol Dweck’s growth mindset (“teach the kids that intelligence isn’t anything innate, just something you build up over time!”), or in this case, a curriculum that teaches parents, teachers, and children about different “temperaments”:
This is a $3 million, federally funded study, in a high-impact journal, the Journal of Educational Psychology, of what is called a “social-emotional learning”– also sometimes known as “character education.” The intervention is, more or less, to teach parents, teachers, and the kids themselves about different “temperaments” (personality types, essentially), and then suggest different behavioral interventions for parents and teachers and the kids themselves to help deal with their particular temperament. The goal is to improve academic achievement by improving self-control and “sustained attention.” Okay…I can sort of get on board with this.
But lets look at what actually happens in the study.
Because funders really want RCTs, the authors conduct the study as a school-level RCT, with half the schools assigned to the intervention of teaching kids about their different temperaments with puppets and then teaching their parents and teachers about different temperaments and the other half…assigned to a supplementary reading comprehension intervention, where parents and teachers are given extra materials and hints for teaching their kids to read. They call this an attention-control condition, and I understand the reason for making it fairly involved (rather than just handing out a brochure or leaving it as a “business as usual condition.”) You can’t do all the baseline measurement and follow-up measurement that they are requiring of the kids (they’re kindergarteners and first-graders, so you can’t just use standardized tests) without offering something in return for the control families.
Then they do their intervention during kindergarten for most of the kids and during first grade for the rest of the kids, and then follow up a total of four times.
So what happens?
During the first two follow ups, during and immediately after the kindergarten intervention, the control condition kids are…much more likely to be able to read and do math, than the intervention condition kids. The difference is about 0.35 SD for reading and about 0.31 SD for math. These may not seem like big effects, but for education evaluations in the US, they are very big, roughly equivalent to two extra months of instruction at this level. From my quick calculations, the T stats (unadjusted for multiple comparisons) are over 3 for all four comparisons (reading and math during the first four follow-ups).
Hooray! Teaching the parents how to teach their kids how to read worked!
Except this isn’t an evaluation of the reading comprehension intervention. It’s an evaluation of the “social emotional learning” intervention, the “teaching kids about different personality types with puppets” thing. So the researchers then proceed to totally ignore the between child differences, and do a complicated multi-level model of within-child growth curves that somehow shows that the reading comprehension intervention didn’t work and the social-emotional learning teach kids about personality with puppets thing works.
The abstract, introduction, and discussion sections are all about how awesome the “teach kids about personality with puppets” thing is, totally ignoring that the kids who got the control condition were the ones that learned to read and do math.
Now, it’s possible the positive results for the reading comprehension intervention were just a selection effect– maybe the parents that consented were more interested in helping their kids read, so those kids were more likely to do better anyways. (Ironically, the fact that the effects fade by the fifth follow up suggest to me that this isn’t the case entirely.) Or, of course, the reading comprehension intervention could have done nothing and the “Teach kids about personality with puppets” intervention could have had a solely negative effect.
But we’re still left with a situation where the actual differences observed are opposite the reported effect, in a $3 million federal study. I count at least five non-academic articles about this study on Google News, all talking about how well the “teach kids about personality” intervention works, particularly for shy kids.