Methods that Matter
Institute trains educaton researchers

Fall, 2008, Volume 30, Number 2

Felix Fernandez (l.) receives a certificate from IES founding director, Grover “Russ” Whitehurst.

Thanks to a grant from the Institute of Education Sciences’s National Center for Education Research, Larry Hedges, an IPR faculty fellow, and Mark Lipsey and David Cordray of Vanderbilt University launched a summer institute on randomized cluster trials in education research. Thirty researchers from around the country attended an intensive two-week, hands-on training session July 7–17 at Northwestern.

“On the whole, medical researchers have far more experience in conducting randomized experiments than education researchers do,” Hedges observed. “After teaching a workshop on how to conduct better randomized-controlled trials (RCTs) in health research, I realized that if well-established
medical researchers had a need for such workshops, then education researchers had an even greater need for the same kind of experimental training.” Hedges is Board of  Trustees Professor of Statistics and Social Policy at Northwestern.

Researchers leave their academic programs schooled in methodological best practices for that time, Hedges continued. But these practices change over time, and little exists in the way of training for those who want to keep abreast of current best practices—especially for midcareer researchers. So Hedges, a national leader in the fields of education statistics and evaluation, set out with colleagues at Vanderbilt and the Institute of Education Sciences to create such an institute.

Bonnie Doren, a senior research associate and assistant professor at the University of Oregon who participated in the workshop, said, “Over the years, education researchers have been struggling to figure out how to do more rigorous evaluations in education settings, and we have been struggling with that as well.”

Doren has spent the last 10 years developing programs for high school students with disabilities and conducting research in secondary special education. She said she decided to apply for the two-week workshop instead of shorter, regional workshops because she felt it was important to step away from a local focus for a broader, more national one “and really get immersed in the topic.” She also wanted to learn from those who had faced some of the same research issues that she had.

	Larry Hedges helps participants work through a data analysis exercise at the IES summer institute.

The program was very intense, Doren said, but she appreciated how the instructors covered not only theoretical elements but practical elements as well, such as what reviewers look for in grant applications.

Felix Fernandez, a consultant with ICF International for the past two years, said the “big names” first drew him to the institute. “These are the foremost experts in the field of RCTs,” he said.

Fernandez said he appreciated that the instructors used actual examples in explaining how to do power calculations, supplying him with knowledge that he could use right away. Back on the job, he was able to quickly and correctly calculate the statistical power for the second year of a randomized trial on which he was working.

Fernandez also found it enriching to interact with a wide variety of people who came to the institute, from academics to research consultants and graduate students. The faculty organizers did an “excellent job” of pairing people who typically do not work together, he said, noting this served to enhance the instruction.

The topics included describing and quantifying outcomes, specifying conceptual and operational models, basic experimental designs for education studies, sampling size and statistical power, and using software like HLM to conduct hierarchical data modeling. Participants also worked on a group project, in which they had to conceptualize and submit a funding application for an education experiment.
“The workshop blew away all my expectations,” Fernandez concluded. “It was a lot more thoughtful and well put together than I would have expected, and it was one of the best trainings I’ve ever attended.”

IES Director Distributes Certificates and Advice

At the end of the workshop on July 17, each of the participants received a certificate of completion from IES founding Director Grover “Russ” Whitehurst, who flew in from Washington, D.C., to make the presentations. He has been at the helm of IES since it was founded in 2002 to provide rigorous evidence on which to ground education practice and policy.

Whitehurst explained that IES “is enthusiastically invested in the summer institute” because of a tremendous deficiency of methodologically rigorous experiments in the field of education.

“There’s much more important work that desperately needs to be done than there are people well-trained to do that work,” he said. “You are sharpening your skills to be able to provide answers that policymakers can stand on.”

The IES director, whose six-year appointment is ending this fall, also spoke of his struggle to make Washington policymakers understand why methods matter and encouraged participants to be just as vigilant about convincing others of the importance of conducting research the right way.

“How can we be sure that social programs are having their intended impact? It’s when you are looking for that answer that methods matter,” Whitehurst said. “We want to provide answers that are the right answers.” Everyone from foundations to policymakers and the public needs to know whether, for example, smaller high schools or smaller classes provide a better education, he said.

Whitehurst reminded participants of what hangs in the balance between a right answer and a wrong answer—the lives of those whom the programs and policies are supposed to serve.

“People who are struggling need education and social programs that work,” he continued. “They don’t need an education industry that continues to provide pseudoservices that, in fact, don’t provide them with the help that they need. … So we need to focus on things that matter and utilize methods that can give us right answers,” Whitehurst concluded.

The institute will be held again next summer at Vanderbilt University. For more information, please visit the Q-Center’s Web page at www.northwestern.edu/ipr/qcenter.