The other day I was reviewing a paper that looked quite interesting, but unfortunately was written in such poor English that I could not really understand what was going on. I felt sorry for the author(s). I then recalled a recent discussion with a colleague of mine about how important so-called transferable skills are for students: We know that most of them won’t end up in academia, so stuff like critical thinking, structuring an argument, or reading a regression table a are pretty important. Among these, coherent and comprehensible English must rank very high. For those who stay in academia, I’d argue that it’s the most important skill, because it’s central to communicating with other researchers and having your work understood. Only this way can others build on what we do. Ironically, however, teaching English is typically not a focus at universities, if it is done at all. Like so many things, we just kind of assume students (have to figure out how to) do it.
Image: CC-by-nc Moiggi Interactive
P-values are hard enough to understand — the appear ‘magically’ on the screen — so how can we best communicate the problem of p-hacking? How about using Yahtzee as an analogy to explain the intuition of p-hacking?
In Yahtzee, players roll five dice to make predetermined combinations (e.g. three of a kind, full house). They are allowed three turns, and can lock dice. Important for the analogy, players decide which of combination they want to use for their round after the three turns. (“I threw these dice, let’s see what combination fits best…”) This is what adds an element of strategy to the game, and players can optimize their expected (average) points.
Compare this with pre-registration (according to Wikipedia, this is actually a variant of the Yahtzee variant Yatzy — or is Yahtzee a variant of Yatzy? Whatever.). This means players choose a predetermined combination before throwing their dice. (“Now I’m going to try a full house. Let’s see if the dice play along…”)
If the implications are not clear enough, we can play a couple of rounds to see which way we get higher scores. Clearly, the Yahtzee-way leads to (significantly?) more points — and a much smaller likelihood to end up with 0 points because we failed to get say that full house we announced before throwing the dice. Sadly, though, p-values are designed for the forced Yatzy variant.
Image: cc-by by Joe King
As part of the course on applied statistics I’m teaching, my students have to try to replicate a published paper (or, typically, part of the analysis). It’s an excellent opportunity to apply what they have learned in the course, and probably the best way to teach researcher degrees of freedom and how much we should trust the results of a single study. It’s also an excellent reminder to do better than much of the published research in providing proper details of the analysis we undertake. Common problems include not describing the selection of cases (where not everyone remains in the sample), opaque recoding of variables, and variables that are not described. An interesting case is the difference between what the authors wanted to do (e.g. restrict the sample to voters) and what they apparently did (e.g. forge to do so). One day, I hope this exercise will become obsolete: the day my students can readily download replication code…
Image: CC-by-nd Tina Sherwood Imaging https://flic.kr/p/8iz7qS
I have just finished grading a batch of reports and am happy to see that my students this year seem to being better. What makes me particularly happy is that there were fewer “serious flaws”.
What a motivating experience I had today: I gave my students some difficult exercises, and they tried hard and did well. I just thought I’d share this while it’s still fresh!