November 9, 2005 | Research

Surveys Miss Racial Discrimination in Employment

By Pat Vaughan Tremmel

EVANSTON, Ill. --- In a survey designed to gauge employment bias, employers showed an equal likelihood of hiring black and white employees. But, in actuality, what employers said they would do and what they did on a previously conducted employment audit were quite different, according to study co-investigator Lincoln Quillian, a professor of sociology and faculty fellow at the Institute for Policy Research at Northwestern University.

To subtly measure bias, the survey questions included race as one of a number of attributes of a hypothetical candidate described in a vignette. The study asked some employers about the likelihood of their hiring a hypothetical white applicant for an entry-level job and others about the likelihood of hiring a black applicant.

Despite the lack of bias shown in the survey, employers were three times more likely to hire a white person than a black person, according to the study “What Employers Say Versus What They Do.” (The study, published in the American Sociological Review, June 2005, was led by Devah Pager, a professor at Princeton University, as well as Quillian.)

The study suggests that surveys of employers, widely used by social scientists and policy people to gauge discrimination, are inherently flawed as measures of actual discrimination.

“Surveys are widely used for social science research in part because they are easy to do and fairly cheap,” Quillian says. “But our research shows just about no association between answers given in this particular survey and the behavior of the employers we audited.”

For the study, both black and white testers, actually college graduates, filled out job applications for entry-level jobs at such places as factories, restaurants and stores. They chose from 350 job listings published in the Milwaukee Journal Sentinel in December 2001. The testers listed nearly identical employment, education and criminal histories on their applications. They also were trained to exhibit similar interpersonal styles to employers.

Later employers, who did not know that they had been audited for the study, were surveyed by researchers to determine whether their words matched their actions. Employers audited by black auditors were asked whether they would hire a black applicant with job-relevant characteristics similar to those of the black auditors they actually interviewed. Likewise, employers audited by white auditors were asked about hiring a similar white applicant.

“The distribution of how likely employers were to hire the applicant was the same regardless of race, though the audits themselves showed quite a different story,” Quillian says. “The surveys failed to reveal any racial bias whatsoever.”

The long, complicated literature about the relationship of survey measures with discrimination in practice has very mixed results, Quillian says.

“Most of the studies that attempt to gauge discrimination set up artificial conditions to examine minor kinds of behavior and miss the context of the real world of an employer actually making a consequential decision about who to hire and not to hire. But this study along with other related audit studies done in recent years show a fair amount of racial discrimination in employment,” he says.

“Walking the Talk? What Employers Say Versus What They Do” also found that employers who indicated a greater likelihood of hiring ex-offenders were no more likely to hire ex-offenders in practice. In addition, employers were three times more likely to hire a white ex-offender as a black one with similar background and qualifications.

For Quillian, the flawed survey measurement is one of the study’s most striking findings. “This study, which matches words with actions, suggests that social scientists and policy makers ought to be very cautious about relying on surveys to measure discrimination in employment.”