How well do correspondence tests measure discrimination?

Correspondence tests are a useful field experiement to measure discrimination in the formal labour market. These correspondence tests are also known as CV experiements: Researchers send two equivalent applications to an employer, differening only in the quantity of interest — gender and ethnicity are common. If only the majority or male candidate is invited for a job interview, we probably have a case of discrimination. Once we aggreate across many employers, we’re pretty confident to have captured discrimination.

Most studies stop there, declining any offer to reduce the burden on employers. The hiring process, however, does not end there. Lincoln Quillian and his team have now compiled a list of studies that went further. They find that the first stage of screening is far from the end of discrimination, and the job interview can increase overall discrimination substantially. Correspondece tests focusing on the first stage will capture only some of the discrimination. Interestingly the discrimination at the job interview stage appears unrelated to discrimination at the first screening of applications.

Quillian, L., Lee, J., & Oliver, M. (2018). Meta-Analysis of Field Experiments Shows Significantly More Racial Discrimination in Job Offers than in Callbacks. Northwestern Workin Paper Series, 18(28). Retrieved from

Zschirnt, E., & Ruedin, D. (2016). Ethnic discrimination in hiring decisions: A meta-analysis of correspondence tests 1990–2015. Journal of Ethnic and Migration Studies, 42(7), 1115–1134.

Image: CC-by Richard Eriksson.

Audit Studies — The Book

There’s a new book edited by S. Michael Gaddis on audit studies. The subtitle promises to go behind the scenes with theory, method, and nuance — and this is what the book provides. As such, the book is a much needed contribution to the literature, where we typically see the results and little how we got there. With (not so) recent concerns around researcher degrees of freedom, the tour behind the scenes offered by the various chapters are an excellent way to make visible and apparent the ‘undisclosed flexibility’ as Simmons et al. called it in 2011. It’s one thing to discuss this in abstract terms, and it’s another thing to sit down with actual research and reflect on the many choices we have as researchers. Indeed, public reflection on research practices may be relatively rare in itself when it comes to quantitative research.

The book comes with a dedicated support webpage: (do me the favour to update the “coming soon” banner). On this website, several chapters can be downloaded as pre-prints, though it’s not all the contents if someone is looking for a free book. I hope the authors will make their code available on the website as promised in several places in the book, because this will be another greatly helpful resource for those new to audit studies or looking for new directions.

I greatly enjoyed to read the reflections by other researchers doing audit studies, and would definitely recommend the book to anyone thinking of doing an audit study. At times there were passages that seemed a bit redundant to me, but all the chapters are written in such an accessible way that this didn’t bother me much. Where I think the book falls a bit short is on two fronts. First, it is very US-centric. In itself this is not an issue, but there are several instances where the authors don’t reflect that perhaps in other countries the markets are not organized the same way. In my view, a comparison to other countries and continents would have been fruitful to underline some of these assumptions — I’ve tried to just this on attitudes to immigrants. Second, the book is not a guidebook. I know, it doesn’t claim to be one, but the book asks so many (justified) questions and offers comparatively few concrete guidelines like Vuolo et al. offer it on statistical power. In this sense, the book will stimulate readers to think about their own research design and not provide a template. And this is actually a good thing, because as the chapters make apparent without normally saying so, there is no universal approach that suits different markets in different places and at different times.

So, should you buy the book? Yes if you want to carry out your own audit study, yes if you want to better understand and qualify the results of audit studies, and yes if you’re looking for guidelines — because the book will make you realize that you’re largely on your own. What would probably useful, though, would be a checklist of things to consider, something readers will have to create themselves on the basis of chapters 4 (Joanna Lahey and Ryan Beasley), 5 (Charles Crabtree), and 6 (Mike Vuolo, Christopher Uggen, and Sarah Lageson).

Gaddis, S. Michael, ed. 2018. Audit Studies: Behind the Scenes with Theory, Method, and Nuance. Methodos 14. New York: Springer.

Ruedin, Didier. 2018. ‘Attitudes to Immigrants in South Africa: Personality and Vulnerability’. Journal of Ethnic and Migration Studies.

Simmons, Joseph P., Leif D. Nelson, and Uri Simonsohn. 2011. ‘False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant’. Psychological Science 22 (11): 1359–66.

Vuolo, Mike, Christopher Uggen, and Sarah Lageson. 2016. ‘Statistical Power in Experimental Audit Studies: Cautions and Calculations for Matched Tests With Nominal Outcomes’. Sociological Methods & Research, 1–44.

Zschirnt, Eva, and Didier Ruedin. 2016. ‘Ethnic Discrimination in Hiring Decisions: A Meta-Analysis of Correspondence Tests 1990–2015’. Journal of Ethnic and Migration Studies 42 (7): 1115–34.>/small>