WP-00-01
Using
Performance Standards to Evaluate Social Programs with
Incomplete Outcome Data: General Issues and Application
to a Higher Education Block Grant Program
Charles F. Manski,
John Newman, and John V. Pepper
Abstract
The basic idea of program evaluation is both simple
and appealing. Program outcomes are measured and compared to some
minimum performance standard or threshold. In practice, however,
evaluation is quite difficult. Two fundamental problems of outcome
measurement must be addressed. The first, which we call the problem
of auxiliary outcomes, is that we do not observe outcome of interest.
The second, which we call the problem of counterfactual outcomes,
is that we do not observe the threshold standard. This paper examines
how performance standards should be set and applied in the face
of these problems in measuring outcomes. In particular, we consider
the problem of evaluating the new World Bank-sponsored Quality
of Undergraduate Education (QUE) program. This competitive block
grant program is to be judged by the program's effects on student
outcomes, not by the particular ways in which grantee departments
use their funds. Our central message is that the proper way to
implement standards varies with the prior information that the
evaluator can credibly bring to bear to compensate for incomplete
outcome data. An evaluator, confronted with the auxiliary and
counterfactual outcomes problems, should combine the available
data with credible assumptions on treatment and outcomes. Given
this information, the performance of a program may be deemed acceptable,
unacceptable, or indeterminate.
Charles F. Manski, Department
of Economics
John Newman, The World Bank
John V. Pepper, Department of Economics, University
of Virginia
