Rothstein, Guinier, and the SAT

A year ago, Lani Guinier wrote an article on Salon  , followed by an interview  claiming that there is only a slight correlation between SAT scores and first year college grades, and basing this claim on Jesse Rothstein’s analyses of University of California students. Guinier writes:

“The test makers do not claim it’s a measure of smartness; all they claim is that success on the test correlates with first-year college grades, or if it’s the LSAT (Law School Admission Test), that it correlates with first-year law school grades.

As I’ll explain later, such a correlation is slight at best. [Follows a long critical description of the history of testing and the SAT]
…This is the testocracy in action, an aristocracy determined by testing that wants to maintain its position even if it has to resort to fabrication. What is it they are so desperate to protect? The answer initially seems to be that the SAT can predict how well students will do in college and thus how well-prepared they are to enter a particular school. There is a relationship between a student’s SAT score and his first-year college grades. The problem is it’s a very modest relationship. It is a positive relationship, meaning it is more than zero. But it is not what most people would assume when they hear the term correlation.

In 2004, economist Jesse Rothstein published an independent study that found only a meager 2.7 percent of grade variance in the first year of college can be effectively predicted by the SAT.”

At the time, many academic friends posted Guinier’s essay on Facebook, and repeated the “weak correlation” and 2.7 percent figure, urging their schools to drop the SAT. I hadn’t read Rothstein’s article, but the 2.7 percent seemed shocking to me. Could measured cognitive ability really be so weakly correlated with college grades? Even if weaker students were selecting into easier majors and schools, that seemed impossible, and I had seen some research that directly contradicted it.

For example, here is a study of the 159,000 entering freshmen in 2007 with valid SATs, HSGPA, and first-year GPAs:

The correlation between the combined three sections of the SAT and first year GPA is r=0.59. In other words, the SAT by itself explains about 35% of the variation in college grades, over ten times as much as Guinier claims. (When combined with HSGPA, the SAT still explains 8% of the variation, still much more than in Guinier’s argument– and it’s not like she even included, “when controlling for high school grades.”) So what gives? Did Rothstein really find that, at UC schools (uniquely), the SAT does not predict grades?

Of course not. Here is the paper.

Rothstein finds that, when controlling for high school grades *and comparing demographically and socioeconomically similar schools*, the SAT has much lower additional predictive validity. There .are a lot of explanations for this, most of which aren’t really that damning of the SAT. The simplest one is that, at a school with a lot of rich kids all of whom are expected to go to college, the teachers feel pressure to give assignments and tests that are closer to what would be expected in college– so the HSGPA by itself is more informative. At poorer schools where very few kids go to college, the few kids who get good SAT scores but bad grades are just not going to be very prepared for college– they didn’t do well even by the low standards of their high school. In either case, the additional predictive validity of the SAT is going to be low, once you know a lot about the kid’s high school and grades.
Looking across high schools (as in the larger study above), thefact that SAT has high predictive validity by itself but lower when combined with HS Grades just suggests that, to a limited degree, the SAT is keeping the schools honest. Without the SAT, high schools would be tempted to inflate grades even more than the insane level they already are, and college admissions would, perhaps, just descend into competitive mythomania.

As for Rothstein– did he just not know that his study would be misrepresented in this way? Or did he know but not care? In the end, politicians will be more influenced by how academic work is represented in the popular press than its original form, so it’s important for academics to clarify when their work is presented incorrectly. And if anybody tells you that the SAT isn’t correlated with college grades- it ain’t true.

One thought on “Rothstein, Guinier, and the SAT

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s