Another use for pre-registration plans (PAP): help yourself

Pre-registration plans (PAP) rightly become more common (they are still not common enough yet, I think), but here’s a reason to write up a PAP that I have never seen mentioned before: Pre-registration plans can be immensely useful for yourself!

So, you have come up with a clever analysis, and writing the PAP has helped sharpen your mind what exactly you are looking for. You then collect your data, finish off another project, and … what was it exactly I was going to do with these data? Did I need to recode the predictor variable? etc.? Yes it happens, and a pre-analysis plan would be an ideal reminder to get back into the project: PAP can be like a good lab journal or good documentation of the data and analysis we do — a reminder to our future selves.


I have recently explored open-source approaches to computer-assisted qualitative data analysis (CAQDA). As is common with open-source software, there are several options available, but as is often also the case, not many of them can keep up with the commercial packages, or are abandoned.

Here I wanted to highlight just three options.

RQDA is built on top of R, which is perhaps not the most obvious choice — but can have advantages. The documentation is steadily improving, making it more apparent how RQDA has the main features we’ve come to expect from CAQDA software. I find it a bit fiddly with the many windows that tend to be opened, especially when working on a small screen.

Colloquium is Java-based, which makes it run almost everywhere. It offers a rather basic feature set, and tags can only be assigned to lines (which also implies that lines are the unit of analysis). Where it shines, though, is how it enables working in two languages in parallel.

CATMA is web-based, but runs without flash — so it should run pretty anywhere. It offers basic manual and automatic coding, but there’s one feature we really should care about: CATMA does TEI. This means that CATMA offers a standardized XML export that should be usable in the future, and facilitate sharing the documents as well as the accompanying coding. That’s quite exciting.

What I find difficult to judge at the moment, is whether TEI will be adopted by CAQDA software. Atlas.ti does some XML, but as far as I know it’s not TEI. And, would TEI be more useful to future researchers than a SQLite database like RQDA produces them?

Discrimination not declining

A new meta-analysis draws on correspondence tests in the US to show that levels of ethnic discrimination in hiring do not seem to have changed much since 1989. This persistence in racial discrimination is bad news, and indeed Eva Zschirnt and I have shown the same result across OECD countries a year ago. While policies have changed, especially in the European Union, looking at the ‘average’ from correspondence tests suggests that they may not have been effective — and that is bad news.

Correspondence tests are widely accepted as a means to identify the existence of ethnic discrimination in the labour market, and as field experiments they are in a relatively good position to make the causal claims we typically want to make. It turns out that most correspondence tests have not paid sufficient attention to heterogeneity, which — as David Neumark and Judith Rich demonstrate — means that they likely over-estimate the degree of discrimination. Unfortunately, most old studies did not vary the groups in a way that this could be fixed post-hoc. If we throw these out of the meta-analysis, we probably no longer have sufficient studies to make claims about changes over time.

Meta-analyses are no doubt an important tool of science, but there’s always a delicate balance to be struck: are the experiments included really comparable? Here we’re looking at field experiments in different countries, different labour markets, different jobs, and different ethnic groups. We can control for these factors in the meta-analysis, but with the limited number of studies we have, this might not be sufficient to silence critics. With correspondence tests, we only cover entry-level jobs, and despite much more fine-graded studies going into the field recently, we don’t have a tool to really identify why discrimination takes place.

Neumark, David, and Judith Rich. 2016. ‘Do Field Experiments on Labor and Housing Markets Overstate Discrimination? Re-Examination of the Evidence’. NBER Working Papers w22278 (May).

Quillian, Lincoln, Devah Pager, Ole Hexel, and Arnfinn H. Midtbøen. 2017. ‘Meta-Analysis of Field Experiments Shows No Change in Racial Discrimination in Hiring over Time’. Proceedings of the National Academy of Sciences, September, 201706255. doi:10.1073/pnas.1706255114.

Zschirnt, Eva, and Didier Ruedin. 2016. ‘Ethnic Discrimination in Hiring Decisions: A Meta-Analysis of Correspondence Tests 1990–2015’. Journal of Ethnic and Migration Studies 42 (7): 1115–34. doi:10.1080/1369183X.2015.1133279.

Image: CC-by CharlotWest

Academic genealogy

There are several projects out there to trace academic genealogy out there, the biggest one is probably the Academic Tree. The idea is to trace who was your supervisor’s supervisor’s … A while ago I looked into what this would look like for me. This being the social sciences, PhD advisors did not exist all the way back. I’m not sure how the Mathematics Genealogy Project go about this: do they include research assistants? (I did so for the ‘ancestors’ of David Glass.)

Funny enough, I’m not quite sure what this means. Sure, your PhD supervisor has a big impact on how you do research and how you see the world, but aren’t we more influenced by what we read and the courses we took before that, for example, and all the research we undertake after that. (I’m not even trying to think of causality here.) Is it at all relevant for me today that one of my academic ‘ancestors’ was quite outspoken against the eugenics movement when it was still quite popular? Whatever.

Supplementary Material for “Estimating Party Positions” on OSF/SocArXiv

I have just made available the supplementary material for Ruedin, Didier, and Laura Morales. 2017. “Estimating Party Positions on Immigration: Assessing the Reliability and Validity of Different Methods.” Party Politics available on OSF/SocArXiv. The supplement is also available at the publisher’s website, together with the article. In the paper, we systematically assess various methods to identify the position political parties take on immigration. In another paper about to be published by Party Politics, Christoffer Green-Pedersen and Simon Otjes demonstrate that immigration really has become more salient over time. All the more important it is to place parties on this issue, and our extensive evaluation finds high consistency between expert surveys, manual sentence-by-sentence coding and manual ‘checklist’ coding. On the other hand, there are inconsistent results with the CMP/MARPOR, Wordscores, Wordfish, and a dictionary approach using keywords.

Green-Pedersen, Christoffer, and Simon Otjes. 2017. “A Hot Topic? Immigration on the Agenda in Western Europe.” Party Politics, doi:10.1177/1354068817728211.

Ruedin, Didier and Laura Morales. 2017. “Estimating Party Positions on Immigration: Assessing the Reliability and Validity of Different Methods”. Party Politics. doi:10.1177/1354068817713122