Workshop Purposes and Structure

The aim of this workshop is to begin the process of improving the quality of quasi-experimental research that takes place in education. To this end, the workshop will be oriented to the strongest quasi-experimental designs for which there is independent evidence of their validity or very strong presumptions thereof. While these are designs and statistical approaches being increasingly used in education, they are still not common. So the workshop will deal with taking advantage of the conditions that facilitate their use.

However, the workshop will deal just as extensively with the most commonly used design in education, the one having two or more non-equivalent groups and a pretest and post-test measure from each. Relying on recent research on the conditions under which experiments and quasi-experiments of this form give the same answer, we will detail the ways in which this design should be implemented and analyzed to achieve the best approximation to experimental results. We will also examine the ways in which the design can be improved by simple and feasible add-ons so as to identify the conditions under which the design should not be used.

Throughout we will use numerous examples, all from education. Before attending the workshop attendees will be sent a book of these articles in case they want to read it. We expect the articles to make more sense after the workshop, though, and trust that the book will remain a reference source long after the workshop is over. There is no formal text, but attendees might do well to have a copy of Shadish, Cook & Campbell, Experimental and Quasi-Experimental Design for Generalized Causal Inference, Houghton-Mifflin, 2002.

Since methods are best learned with concrete examples, we encourage attendees to bring with them details of quasi-experimental projects or quasi-experimental theory issues they are working on. There will be opportunity for one-on-one time with the instructors.

Day 1

9:00am-12:00pm: Introductions

•

•

•

•

•

•

Purposes, people and schedule (1 hour)

Two concepts of causation—activity and explanatory theories

Rubin Counterfactual Model and its links to random assignment

The Pattern Matching Model and its links to multiple implications

Four types of validity very briefly described

Validity priorities for this workshop

Lunch: 12:00pm—1:00pm

1:00pm-4:00pm: Randomized Experiment with Individuals and Clusters

•

•

•

•

•

•

•

Getting consent

Ethics

Proper and improper randomization

Systematic attrition

Treatment contamination and using instrumental variables to model it

The Theoretically Unnecessary but Pragmatically Necessary Pretest

Special issues with cluster-level experiments:

° statistical power and computing sample sizes

° small sample sizes and unhappy random assignment

° dealing with treatment misassignment

° analysis using multi-level models

Please attend dinner with Tom and Will

Day 2

9:00am-12:00pm: Regression Discontinuity: The Basics

•

•

•

•

•

•

•

What is the design?

Why it is unbiased

Many examples from education

Modeling functional form in psychology and in economics

Fuzzy discontinuities and instrumental variables

Weighting at the cut-off point

Statistical power considerations

Lunch: 12:00pm-1:00pm

1:00pm-4:00pm: Regression Discontinuity: Beyond the Basics

•

•

•

Adding additional design elements, with examples

Hypothetical example of evaluating No Child Left Behind sanctions

How to get the design used more often

Open invitation to all who would like to dine with Tom and Will

Day 3

9:00am-12:00pm: Abbreviated ITS Designs and Analysis

•

•

•

•

•

•

Many reasons for adding more pretest time points

Design of simple ITS, with education examples

Identifying usual threats to internal and construct validity

Dealing with these threats within the simple ITS framework

Adding design elements to improve interpretation, with examples

How to analyze the data given non-independent observations

Day 4

9:00am-12:00pm: "Workhorse Design"—Pre/Post with Non-Equivalent Groups

•

Illustrating the design and the usual analytic problems with it

•

Bad matching and statistical regression

•

Better population matching through initial sampling design: Bloom et al., & Aiken et al.

•

Knowing and measuring the selection process; and knowing and modeling the outcome: Shadish et al.

•

Other local matching techniques for education: Twin, sibling, grade and cohort control matches

•

The general principle is ... ?

Lunch: 12:00pm-1:00pm

1:00pm-4:00pm: Statistical Analysis of "Work Horse Design"

•

•

•

•

•

•

•

Ordinary least squares, specification bias and errors in pretest

Heckman-type selection models

Propensity scores

Instrumental variables within an experriment vs. a substitute for it

Level of analysis as a complicating factor—student, class, school

How interpretation depends on design and measurement as well as analysis

Designs with better and worse covariate features

Please attend dinner with Tom and Will

Day 5

8:30am-11:30am: Beyond Statistical Matching as Model for Selection Control—Design Elements to add to the basic "Workhorse Design"

•

•

•

•

•

•

•

Cohorts as non-equivalent controls, examples

More pretest waves, examples

Multiple control groups, examples

Non-equivalent dependent variables, examples

Uses (and abuses) of variation in treatment implementation

Simultaneous use of several such design elements

This same logic applied to within-study designs

Lunch: 11:30am-12:30pm

12:30pm-2:30pm: Common Designs to Avoid, and Wrap-Up

•

•

•

•

•

•

•

•

Designs with no pretest, but how can we then do kindergarten studies?

Limited designs with no control groups

Proxy pretests and also

Instrumental variables

Pattern matching designs with no control groups—Minton as the example

Generalizing Minton

Unexamined issues attendees want to discuss

Keeping in touch

3.

Individual Consultations

We hope that you will come to this workshop with specific quasi-experimental projects in mind or already underway or with a concern for theoretical issues surrounding quasi-experimentation.

There is the possibility of discussing these projects with either Tom Cook or Will Shadish either before or after class sessions or during them when one of the instructors is leading the class and the other is free to consult with you.

We encourage this one-on-one interaction that is grounded in a specific quasi-experimental project or theoretical concern. We want to make time for this, not just because it is important to you, but also because we learn by learning about problems you have that are not amenable to obvious solution. So please bring these issues to our attention. We anticipate being able to help you, and we know you will help us.