Why We Habitually Engage in Null-Hypothesis Significance Testing…

You should head over to PLOS to read this paper by Jonah Stunt et al. It’s the first qualitative study I’ve come across at PLOS, but it’s definitely worth a read to better understand why we’re still surrounded by p-values.

One thing I missed in the paper is a hint that we don’t have to engage in frequentists null-hypothesis significance testing. I realize that the authors are interested in the sociology of science here, but we have plenty of statements in the article how difficult it’d be to learn about alternative methods. It doesn’t have to be: We do have packages like rstanarm or software like JASP that do not leave much room for such excuses.

Stunt, Jonah, Leonie van Grootel, Lex Bouter, David Trafimow, Trynke Hoekstra, and Michiel de Boer. 2021. “Why We Habitually Engage in Null-Hypothesis Significance Testing: A Qualitative Study.” PLOS ONE 16(10):e0258330. doi: 10.1371/journal.pone.0258330.

Living Together or Side by Side? New Study out now!

Our new study examines how residents in Switzerland perceive migration-related social change in their municipality, their place of work, and in public. We left the ivory tower and listened. The result is a detailed and diverse picture: Migration is perceived as part of social change more widely, but it’s not migration as such that evokes threat. Perceptions of threat and fear are a side-effect of wider social change and economic growth, such as changes to the built environment because of new buildings, cars and transportation, and a perceived impoverishment of social life. It is clear that a majority seek communities with local opportunities to meet and exchange, but many also recognize that the world changes.

The report is available in French (https://www.ekm.admin.ch/ekm/fr/home/dokumentation/studien.html), German (https://www.ekm.admin.ch/ekm/de/home/dokumentation/studien.html), or Italian (https://www.ekm.admin.ch/ekm/it/home/dokumentation/studien.html).

Doing Fieldwork in a Pandemic

Here’s a document that deserves more attention: doing fieldwork in a pandemic, a crowd-sourced document initiated by Deborah Lupton.

Isolation measures to contain the spread of COVID-19 means that social researchers who conduct face-to-face fieldwork (interviews, focus groups, participant observation, ethnographies etc) are now faced with the challenge of either delaying or re-inventing their methods so that they can continue their research until these measures are relaxed.

Doing fieldwork in a pandemic

CATMA does TEI

I have recently explored open-source approaches to computer-assisted qualitative data analysis (CAQDA). As is common with open-source software, there are several options available, but as is often also the case, not many of them can keep up with the commercial packages, or are abandoned.

Here I wanted to highlight just three options.

RQDA is built on top of R, which is perhaps not the most obvious choice — but can have advantages. The documentation is steadily improving, making it more apparent how RQDA has the main features we’ve come to expect from CAQDA software. I find it a bit fiddly with the many windows that tend to be opened, especially when working on a small screen.

Colloquium is Java-based, which makes it run almost everywhere. It offers a rather basic feature set, and tags can only be assigned to lines (which also implies that lines are the unit of analysis). Where it shines, though, is how it enables working in two languages in parallel.

CATMA is web-based, but runs without flash — so it should run pretty anywhere. It offers basic manual and automatic coding, but there’s one feature we really should care about: CATMA does TEI. This means that CATMA offers a standardized XML export that should be usable in the future, and facilitate sharing the documents as well as the accompanying coding. That’s quite exciting.

What I find difficult to judge at the moment, is whether TEI will be adopted by CAQDA software. Atlas.ti does some XML, but as far as I know it’s not TEI. And, would TEI be more useful to future researchers than a SQLite database like RQDA produces them?

Qualitative Studies are often not quite as small N

Qualitative studies are often described as small N studies because the number of respondents is small. I argue that this is the wrong perspective: What we really have in qualitative data, say interviews, is lots of data (points) clustered within individuals. Rather than focusing on the number of respondents, we should probably focus on the number of relevant statements (i.e. statements about our quantity of interest), and describe this number (along with the number of respondents). When computer aided qualitative data analysis (CAQDA) is used, I guess the number of tags relevant to our quantity of interest is that number. Seen this way, many qualitative studies are no longer small N studies, but we’re still faced with unstructured, messy data that may be difficult to analyse, and of course we don’t have independent observations — so generalization remains a challenge.