A good scientific theory starts with regularly observable phenomena; then you get an accurate description of the patterns underlying the phenomena; then ideally you get the lasting rules that drive those patterns.
First you get Babylonian astronomers up through Tycho Brahe, accurately recording what they see in the sky, then you get Kepler describing the structure of the cosmos that would produce those patterns, then Newton explaining the laws behind why that structure would work the way it did. It took a while.
The problem for social scientists is they want it all at once. Rather than worrying about the general, boring, repeated patterns of social phenomena, they want a novel, unusual phenomenon (that’s what the Diedrick Stapel fakery was all about), that thanks to the supposed magic of an RCT or a clever quasi-experiment reveals the causal agent producing that phenomenon. Even more than that, they want something actionable, a causal agent that Somebody can Do Something About, like how all the Michael LaCour fakery was about changing people’s minds for good (and for Good) with a quick in-person visit.
On second thought, it’s not so much that social scientists want to skip Brahe and Kepler and go straight to Newton and Einstein. It’s that they want to flap their arms to get to the moon.
With psychometrics, we have years upon years of well-measured data, that fit into regular patterns, and that we are beginning, perhaps, to identify causal agents behind. George W. Bush’s No Child Left Behind law meant that the test scores (technically, proficiency rates) by race and English Language Learner status had to be published for practically every school in the country, so it’s no longer just a matter for researchers: somebody or other in every school district has to frown imposingly at the data every year and create whole separate administrative departments devoted to Closing the Gaps.
But anything actionable in anything more than a folk wisdom sense is probably a ways off. The George W Bush Education Department created a separate Institute for filtering through all the terribly conducted education studies and funding ones that would have some credibility. Then, the idea was that this would reveal that certain practices Work to Close the Gap. But, even ignoring publication bias or potential researcher conflict of interest, there are remarkably few studies that show any credible evidence behind one program or another at all. When the Education Department actually summarizes all the studies together to find what they say about, how to teach Algebra, the evidence behind their suggestions is, they have to admit,”minimal.” The Iron Law (that nothing makes a difference) remains the Iron Law.
The interventions with more evidence behind them tend to be things like KIPP, that are far too intensive for most families to buy into or be universally applied, and that work in part by drawing out the subset of black and Hispanic kids who are ready to commit to round-the-clock schooling, kicking out the kids who aren’t, and avoiding the lawsuits and Justice Department investigations that would be triggered if you tried to apply their disciplinary methods to regular schools. Other plausibly effective interventions are left on the shelf because they go directly opposite how we want things to be: HBCUs might well be more effective for most black college students than UNC or Yale, but nobody who matters wants to hear that.
The stars are up there for all to see (even if it’s harder to see them in Midtown Manhattan), and that certainly was part of why astronomy had a big head start on other sciences. One of the funny things about our era is that an awful lot of educational data has been left out for all to see and come up with their own conclusions from, even if it differs from the Official Line.
It seems at least as plausible to me that the data will stop being available as that the Official Line will change.