Good, Better, Best in Educational Research
Workshops teach best practices in quasi-experimental designs

Spring 2007, Volume 29, Number 1

A group of workshop participants

To help those in educational research circles understand, design, and conduct better quasi-experiments, IPR Faculty Fellow Thomas D. Cook, Joan and Sarepta Harrison Chair in Ethics and Justice, and his colleague William R. Shadish of the University of California, Merced received funding from the Spencer Foundation to launch a series of workshops. They began last spring and will continue this summer.

More than 600 people applied for a slot in one of three week-long sessions last year. The 84 participants came mainly from universities and school districts, but also from the federal government, contract research firms, and research laboratories.

Michael McPherson, president of the Spencer Foundation, noted that the foundation provided funding because the workshops align with Spencer’s aim to strengthen the quality of educational research and, by extension, to improve children’s lives. The workshops provided a “great opportunity” to reach beyond the traditional research realm to those in local school districts and state education circles as well, he said.

Experiments in Education
For many years, randomized clinical trails (RCTs) have been held up as the gold standard of evidence-based research in medicine, agriculture, and other fields—with the exception of education. Recently, however, there has been a large push by the Institute of Education Sciences, which is housed within the U.S. Department of Education, to encourage greater use of RCTs.

“I’m a great fan of that,” Cook noted. “But there are many circumstances where you cannot do a randomized experiment, and generally, the quality of non-experimental research to examine ‘what works’ is low in education.”

RCTs differ from quasi-experiments in that they assign participants randomly to treatment and control groups. However, there are cases where such assignments might not be practical or even feasible, Cook said.

In their workshops, Cook and Shadish paid particular attention to covering the most empirically viable quasi-experimentation practices such as regression-discontinuity designs (RDDs) and interrupted time series. RDDs can also provide unbiased causal estimates because researchers can assign participants to treatment and control groups based on whether they fall above or below a cutoff score, Shadish said.

The two workshop organizers lectured on theory and practice, supplementing their discussions with as many examples as possible from education. In addition to covering the advantages of using such practices, they also highlighted the particular circumstances under which these designs would not work.

What really made the workshop special, according to Cook and Shadish, was their reliance on empirical research that compared the results of randomized experiments to quasi-experiments that shared the same intervention group. Therefore, the experiments only varied in the way in which the control group was formed.

Cook said that they used such examples to counteract a widespread “pessimism” about whether non-experiments could work at all. This attitude apparently grew out of economic studies—mainly job-training reviews—that have compared the findings of randomized experiments with those of individual quasi-experiments. The labor economists’ results seem to indicate that experiments and non-experiments produce different causal estimates, Cook said. Cook and Shadish specifically chose the examples for their workshop to point out the fallacies behind such thinking, and thus, show why and when quasi-experimentation can work.

Tracy Rimdzius, who is an education research analyst at the National Center for Education Evaluation, said she found such examples “comforting”—especially as she is in charge of a study, Reading First, where they recently implemented an RDD. The workshop made her feel more confident about obtaining “rigorous and believable results” from the design.

Rimdzius also found the sessions on interrupted time series designs informative. “There’s a lot of that in education because there is a lot of existing information from state tests,” she said. “It was very enlightening for me to see that even an abbreviated interrupted series requires a lot more data points than people usually have.”

Thomas D. Cook and Spencer Foundation President Michael McPherson review some points of "workhorse designs."

Rimdzius, who also reviews study proposals, noted that she is not quite sure how useful a model it is, given recent state testing changes. It will be some years before enough data points are available to conduct these types of analyses in some states, she said.

Ashaki Coleman,  who coordinates research and assessment for the Cherry Hill Public School District in suburban New Jersey, said that the workshop has given he r a new perspective on her work, especially because of RDD.

Coleman recalled that she had searched long and hard for a technique that would help her to isolate the effects of individual programs within the schools.

“I knew something was out there, and I kept asking people about it. But no one seemed to know much more about it than the name,” Coleman said. “When we learned about RDD in the workshop, I felt like that was the one technique that would just help me tremendously.”

Plus, participants appreciated having the time to discuss their individual projects.

“It was really nice that we were not just talking about the theory of these designs,” Rimdzius said, “but also that we had time to address questions about our particular studies with Tom and Will and each other.”

McPherson, who attended one of the workshop sessions, remarked on how the workshops are practically grounded.

“The great thing about Tom and Will’s workshop is that it’s not about elaborate formal calculations. It’s about sound reasoning,” he said. “Sound reasoning, quite frankly, is too often missing in social science work. People just don’t think it through, and I think Tom and Will are great at doing that and showing how it’s going to be done in the context of quasi-experiments.”

The workshops were also learning experiences for Cook and Shadish, who had people present and discuss their projects when they overlapped with the curriculum.

“So we weren’t just teaching at them,” Cook said. “We were also identifying issues that warrant further research and are germane to people in educational research. It keeps us from becoming too abstract.”

Cook and Shadish also hope that the workshops will help propagate best practices in quasi-experimentation.

“We brought in a wide variety of people already doing causal research who are in a position to spread the word about better quasi-experimentation practices throughout the world of education,” Shadish said.

Three new workshops, with renewed support from the Spencer Foundation, will take place over the summer, starting in June. For more information, go to www.northwestern.edu/ipr/events/workshops/qeworkshop.html.