Quantitative Methods for
Policy Research

Most researchers and academics tend to stick with the research methods they know best, learned mainly in graduate school—even though those methods might not represent current best practices or the most appropriate method. This is one reason why IPR Faculty Fellow Larry V. Hedges, with the support of a group of distinguished interdisciplinary scholars, launched the Center for Improving Methods for Quantitative Policy Research, or Q-Center, at the Institute for Policy Research. Hedges, who is Board of Trustees Professor of Statistics and Social Policy, co-directs the center with Thomas D. Cook, Joan and Sarepta Harrison Chair in Ethics and Justice. Q-Center faculty work on:

Overview of Activities

Methodology and Research Designs
Economist Charles F. Manski continues his line of work on the difficulties of selecting the best policy with limited knowledge of policy impacts. Manski, Board of Trustees Professor in Economics, finished his forthcoming book Identification for Prediction and Decision (Harvard University Press) that exposits his new methodology for analyzing empirical questions in the social sciences. He recommends that researchers first ask what can be learned from data alone and then ask what can be learned when data are combined with credible weak assumptions. Inferences predicated on weak assumptions, he argues, can achieve wide consensus, while ones that require strong assumptions almost inevitably are subject to sharp disagreements.

Thomas D. Cook, Joan and Sarepta Harrison Chair in Ethics and Justice, conducted a review of the history of the regression discontinuity design (RDD) in psychology, statistics, and economics that will appear in the Journal of Econometrics. Donald T. Campbell, who invented the design in 1958, and a group of Northwestern University colleagues, including Cook, worked on RDD until the early 1980s when the design fell into disfavor. Cook speculates on why RDDs held such a low profile until the mid-1990s. Since then the design has widely caught on, particularly among younger econometricians and labor economists in both the United States and Europe. Cook suggests why this 50-year-old design, rarely used until the beginning of this century, has been reborn.

Cook and IPR graduate research assistant Vivian Wong have published a paper reviewing whether regression-discontinuity studies reproduce the results of randomized experiments conducted on the same topic. They enumerate the general conditions necessary for a strong test of correspondence in results when an experiment is used to validate any non-experimental method. They identify three past studies where regression discontinuity and experimental results with overlapping samples were explicitly contrasted. By criteria of both effect sizes and statistical significance patterns, they then show that each study produced similar results. This correspondence is what theory predicts. But to achieve it in the complex social settings in which these within-study comparisons were carried out suggests that regression discontinuity results might be more generally robust than some critics contend.

Research Methods for Education
Larry V. Hedges, Board of Trustees Professor of Statistics and Social Policy, is reanalyzing surveys with nationally representative samples to develop reference values of intraclass correlations. These data can then be used to help plan experiments in education. For example, one study with graduate student Eric Hedberg provides a compilation of intraclass correlation values of academic achievement and related covariate effects that could be used for planning group randomized experiments in education. This project has funding from the Interagency Educational Research Initiative (IERI). IERI is a collaborative effort of the National Science Foundation, Institute of Education Sciences (IES), and National Institute of Child Health and Human Development to support scientific research that investigates the effectiveness of educational interventions in reading, mathematics, and the sciences.

In another project supported by IES, Hedges is developing improved statistical methods for analyzing and reporting multilevel experiments in education. He is also working on more efficient designs for such experiments that require the assignment of fewer schools. Such designs should reduce the costs of educational experiments and thus make them more feasible to conduct.

For those designs involving cluster randomization, Hedges has defined three effect sizes—and computing estimates of those effect sizes and their standard errors—from information that is likely to be reported in journal articles. He has also provided a simple correction to the t-statistic that would be computed if clustering were incorrectly ignored.

Social psychologist Thomas D. Cook and William Shadish of the University of California, Merced, held three one-week workshops in summer 2006 for 84 educational researchers mainly from universities, contract research firms, and school districts. They will hold three more in 2007. All of the workshops are supported by the Spencer Foundation.

In these workshops, the two organizers cover the most empirically viable quasi-experimental practices such as regression discontinuity designs and interrupted time series. They lecture on theory and practice, supplementing their discussions with as many examples as possible from education, highlighting the advantages and disadvantages of using them. They also rely on empirical research that compares the results of randomized experiments to quasi-experiments that shared the same intervention group.

In addition, Cook continues work on quasi-experimentation in education that is supported by the Spencer Foundation. Many researchers believe that randomized experimentation is usually the best methodology for investigating issues in education. However, it is not always feasible. The usually advocated alternative—quasi-experimentation—has recently come under attack from scholars who contrast the results from a randomized experiment and a quasi-experiment on the same topic, where the quasi-experiment shares the same intervention as the experiment. Thus, the quasi-experiment and the experiment vary only in whether the control group is randomly formed or not. Cook is critically examining this literature, which consists of more than 20 studies.

Research methodologist Spyros Konstantopoulos discusses power analysis in field experiments that involve nested structures where, for example, either entire groups, such as classrooms or schools, or individuals within groups, such as students, are assigned to treatment conditions. The studies illustrate power computations of tests of the overall treatment effect, as well as of tests of the inconsistency of the treatment effect across clusters such as classrooms and schools.

Konstantopoulos’ studies provide methods for computing power of tests of the treatment effect and its variability in three-level designs with two levels of nesting, where for example, students are nested within classrooms, and classrooms are nested within schools. These methods can be extended to quasi-experimental studies that examine group differences in an outcome, associations between predictors and outcomes, and their variability across clusters.

Statistical Accuracy and Forecasting
The accuracy of social statistics is a focus of statistics professor Bruce Spencer’s work. Spencer has started a new project looking at the accuracy of jury verdicts. In a set of 271 cases from four areas, juries gave wrong verdicts in at least one out of eight cases, Spencer found. Based on his findings from this limited sample, he is optimistic that larger, carefully designed statistical studies could tell much more about the accuracy of jury verdicts. If such studies were conducted on a large scale, Spencer believes they could lead to better understanding of the prevalence of incorrect verdicts—false convictions and false acquittals. The IPR working paper “Estimating the Accuracy of Jury Verdicts” recently appeared in the Journal of Empirical Legal Studies.

Data Centers
Q-Center faculty are involved in two major centers for developing data sources, the Data Research and Development Center and the Chicago Census Research Data Center.

The Data Research and Development Center’s ongoing research agenda is to develop and apply research methods for identifying educational interventions that can be scaled up without diminishing the effectiveness of these interventions. The work involves basic research on the design and analysis of studies for determining if an intervention has been scaled successfully, providing technical assistance to similar studies at the Interagency Education Research Initiative (see p. 44). Statistician Larry V. Hedges and Barbara Schneider of the University of Chicago direct the data research center.

Northwestern University is also part of the consortium running the Chicago Census Research Data Center. It provides researchers an opportunity to engage in approved projects using Census Bureau microdata. Other consortium members include Argonne National Laboratory, the Federal Reserve Bank of Chicago, the University of Chicago, and the University of Illinois at Chicago. The center is also supported by a grant from the National Science Foundation. Statistician Bruce Spencer has played a leading role in integrating the center at Northwestern.

Promoting the Methodological Community
Statistician Larry V. Hedges and social psychologist Thomas D. Cook are active in fostering the methodological community at a national level as founding members of the Society for Research on Educational Effectiveness, which held its first meeting in December 2006. The organization seeks to advance and disseminate research on the causal effects of education interventions, practices, programs, and policies.

Hedges and Barbara Foorman of Florida State University are the inaugural editors of the organization’s Journal of Research on Educational Effectiveness. The first issue will appear in early 2008.

Hedges and Cook are also founding members of the Society for Research Synthesis Methodology, a new professional society concerned with the statistical methods for evidence-based social and health policy.

Hedges gave the keynote address, “Meta-analysis at Age 30 (or 102 or 201),” at its first meeting in Cambridge, U.K., in August 2006.

Training New Scholars
The Q-Center has created a four-year postdoctoral training program with funding from the Institute of Education Sciences. The program, which provides funding for two-year fellowships, aims to train postdoctoral fellows in applied education research and produce a new generation of education researchers dedicated to solving the pressing challenges facing the American educational system through methodologically rigorous and relevant research.

Q-Center Colloquia
The first Q-Center colloquia commenced in 2006. Social psychologist Thomas D. Cook gave the inaugural talk on “Observational Studies That Do and Do Not Recreate the Results of Yoked-Randomized Experiments: Making Sense of the Literature in Economics, Education, and Psychology” on March 7.

Economist Charles F. Manski talked about “Fractional Treatment Rules for Social Diversification of Indivisible Private Risks” on April 11.