Academic age

In principle, academic age is simple: the number of years since you completed the PhD. The idea is that your physical age is irrelevant when we evaluate your track records (publications, citations). Instead, we consider how much time you had to build that track.

So far, I guess we’re all on board. However, just looking at the time since your PhD (is that the viva or the graduation ceremony, by the way?) is inadequate if you could not actually do research, like if you were working in industry or engaged in clinical activities (i.e. non-academic employment) or were on parental leave. We can account for this, like the new SNSF CV do, in the spirit of the DORA declaration:

The net academic age is the amount of time you have actually been able to dedicate to research, after deducting interruptions and non-scientific work.

The SNSF template offers: maternity; paternity, adoption, parental leave [as a separate category]; illness or accident; care duties; public service; continuing education; non-academic employment or clinical activities; part-time work; unemployment; other.

However, the more I think about it, the more complicated it gets. For instance, it is increasingly common for PhD students to do paper dissertations, that is, they write a series of academic journal articles rather than a monograph. In terms of research undertaken, this is equivalent, but in terms of publications and citations, a paper dissertation gives you a head start over a classic monograph that may be published a year or so after graduation.

Parental leave and reduced percentages for care reasons seem pretty straight-forward. But what about those who keep working on their research projects during parental leave, do they get a deduction because technically they were on leave, or not because effectively they worked?

Where does academic employment end (or is scientific work the relevant concept? — the guidelines are vague)? In my case, I do a substantial part of my work consulting on commissioned work for government offices etc. Is this academic employment because it entails research and (basic) scientific methods, is this academic employment because the job is at a university (but what if I did the same work in industry, like many of our competitors?), or is this non-academic employment because we produce grey literature (reports that may or may not be published, no peer-review), or because we do not choose our own research question, or because these publications do not count (or even count negatively!) when it comes to tenure prospects?

How do we account for stability? A string of short-term jobs where you spend much of your time applying for the next opportunity are a different beast than tenure-track or tenured positions! And indeed, even two seemingly stable, tenured jobs can be quite different in terms of teaching load. If your net academic age does not consider the fact that some academics can focus all their energy on research, while others are teaching staff doing as much research on the sides as they can, the aim of “allowing us to make a fair comparison with other applicants” is a hollow promise.

I don’t know, we can also add administration and management duties, a boss who gives you the freedom to undertake whatever research you want (or no boss at all) versus a boss who tells you exactly what to work on (e.g. in a project), the resources you have available like research assistants or money to field work or experiments, if authorship practices in your team differ from common practice in your discipline…

And then we get evaluated by criteria like “excellence” (for grants) or “fit” (for jobs) that are as vague as they are explicit (as DORA requires) — does it really make things better if we adjust for net academic age?

Most important academic skill? English!

The other day I was reviewing a paper that looked quite interesting, but unfortunately was written in such poor English that I could not really understand what was going on. I felt sorry for the author(s). I then recalled a recent discussion with a colleague of mine about how important so-called transferable skills are for students: We know that most of them won’t end up in academia, so stuff like critical thinking, structuring an argument, or reading a regression table a are pretty important. Among these, coherent and comprehensible English must rank very high. For those who stay in academia, I’d argue that it’s the most important skill, because it’s central to communicating with other researchers and having your work understood. Only this way can others build on what we do. Ironically, however, teaching English is typically not a focus at universities, if it is done at all. Like so many things, we just kind of assume students (have to figure out how to) do it.

Image: CC-by-nc Moiggi Interactive

Understanding p-hacking through Yahtzee?

P-values are hard enough to understand — the appear ‘magically’ on the screen — so how can we best communicate the problem of p-hacking? How about using Yahtzee as an analogy to explain the intuition of p-hacking?

In Yahtzee, players roll five dice to make predetermined combinations (e.g. three of a kind, full house). They are allowed three turns, and can lock dice. Important for the analogy, players decide which of combination they want to use for their round after the three turns. (“I threw these dice, let’s see what combination fits best…”) This is what adds an element of strategy to the game, and players can optimize their expected (average) points.

Compare this with pre-registration (according to Wikipedia, this is actually a variant of the Yahtzee variant Yatzy — or is Yahtzee a variant of Yatzy? Whatever.). This means players choose a predetermined combination before throwing their dice. (“Now I’m going to try a full house. Let’s see if the dice play along…”)

If the implications are not clear enough, we can play a couple of rounds to see which way we get higher scores. Clearly, the Yahtzee-way leads to (significantly?) more points — and a much smaller likelihood to end up with 0 points because we failed to get say that full house we announced before throwing the dice. Sadly, though, p-values are designed for the forced Yatzy variant.

Image: cc-by by Joe King

Replication as learning

As part of the course on applied statistics I’m teaching, my students have to try to replicate a published paper (or, typically, part of the analysis). It’s an excellent opportunity to apply what they have learned in the course, and probably the best way to teach researcher degrees of freedom and how much we should trust the results of a single study. It’s also an excellent reminder to do better than much of the published research in providing proper details of the analysis we undertake. Common problems include not describing the selection of cases (where not everyone remains in the sample), opaque recoding of variables, and variables that are not described. An interesting case is the difference between what the authors wanted to do (e.g. restrict the sample to voters) and what they apparently did (e.g. forge to do so). One day, I hope this exercise will become obsolete: the day my students can readily download replication code…

Image: CC-by-nd Tina Sherwood Imaging https://flic.kr/p/8iz7qS

Students are getting better = happy

I have just finished grading a batch of reports and am happy to see that my students this year seem to being better. What makes me particularly happy is that there were fewer “serious flaws”.