Ten Questions for Evaluating (Randomized) Evaluations (of Social Programs)
- Was randomization really random?
“Using the administrative database to form the majority of the control group allowed a higher percentage of referrals from high-needs referral sources such as the courts to receive the program.”
“Establishing relationships with intended beneficiaries was critical, as 22 out of 200 program members attended at least one training session.”
“While the central intake system established by the state filtered out individuals who had been placed on a list of study group members, it was impossible to determine if control group members self-enrolled in services themselves or through other referral sources separately.”
“Over 220 of the 400 individuals randomly assigned eventually completed the 1 year follow-up survey.”
5. How non-random or differential was the attrition?
“This follow-up data included all 22 of the program group members who attended at least one training, as well as 198 out of 200 control group members.”
6. If two groups are shown to be comparable at baseline, are they the same two groups for which the alleged impacts are being shown?
“Randomization was successful, with program group members (n=232) scoring on average at the 32nd percentile on the Dweazil-Zappa III 1st Grade Assessment and control group members (n=234) at the 33rd…the nationally normed Moon-Unit-Zappa assessment used in the 5th grade showed that program group members (n=107) scored at the 47th percentile and control group members (n=85) at the 35th, a dramatic gain for the program group.”
7. Any weird confounds?
“Classroom observations for the program schools were conducted by Author A and classroom observations for the control group schools were conducted by Author B.”
“In spite of declining enrollment state-wide, officials were able to identify seventeen sites which they deemed were likely to maintain high enrollment in spite of the challenges of random assignment.”
“Although the program group showed slight declines in outcomes when viewed naively, when properly adjusted using growth-curve analysis it is clear that the Watching Puppet Plays About Feelings intervention outscored the Reading to Your Kids intervention by a statistically significant amount, particularly among high-shyness personality subgroups.”
“As few of you know, I was born Kal-El on the planet Krypton, before being rocketed to Earth as an infant. The yellow star Sol has given me a number of powers unfamiliar to the citizens of Earth, among them the ability to personally collect perfectly normally distributed data with a 96 percent response rate from thousands of respondents, without any outside grant funding.”