DORA … are we getting there?

Yes, we probably all agree that we should evaluate research quality and not quantity. DORA works in that direction, but it avoids specifying what quality means. Perhaps we can even trust each other to identify ‘quality’ and ‘excellence’ just like that.

But consider the following guidelines:

  • “The total number of publications or the number of publications per year is not considered to be the only indicator of performance.”
  • “Each applicant may list up to 10 scientific publications.”

Both of these are attempts to put DORA into practice. In the former, the number of publications per year cannot be used as the sole indicator (“not the only indicator”). In the latter, we actually remove the possibility to do the former (unless we’re evaluating researchers with fewer than 10 outputs).

I don’t know… I’m not convinced we’re changing much other than how we structure CVs and what we highlight. And thinking about it prospectively (early career researchers; planning what research to focus on), can we even guess which research (output) will have a “big” impact on other researchers or society?

Academic age

In principle, academic age is simple: the number of years since you completed the PhD. The idea is that your physical age is irrelevant when we evaluate your track records (publications, citations). Instead, we consider how much time you had to build that track.

So far, I guess we’re all on board. However, just looking at the time since your PhD (is that the viva or the graduation ceremony, by the way?) is inadequate if you could not actually do research, like if you were working in industry or engaged in clinical activities (i.e. non-academic employment) or were on parental leave. We can account for this, like the new SNSF CV do, in the spirit of the DORA declaration:

The net academic age is the amount of time you have actually been able to dedicate to research, after deducting interruptions and non-scientific work.

The SNSF template offers: maternity; paternity, adoption, parental leave [as a separate category]; illness or accident; care duties; public service; continuing education; non-academic employment or clinical activities; part-time work; unemployment; other.

However, the more I think about it, the more complicated it gets. For instance, it is increasingly common for PhD students to do paper dissertations, that is, they write a series of academic journal articles rather than a monograph. In terms of research undertaken, this is equivalent, but in terms of publications and citations, a paper dissertation gives you a head start over a classic monograph that may be published a year or so after graduation.

Parental leave and reduced percentages for care reasons seem pretty straight-forward. But what about those who keep working on their research projects during parental leave, do they get a deduction because technically they were on leave, or not because effectively they worked?

Where does academic employment end (or is scientific work the relevant concept? — the guidelines are vague)? In my case, I do a substantial part of my work consulting on commissioned work for government offices etc. Is this academic employment because it entails research and (basic) scientific methods, is this academic employment because the job is at a university (but what if I did the same work in industry, like many of our competitors?), or is this non-academic employment because we produce grey literature (reports that may or may not be published, no peer-review), or because we do not choose our own research question, or because these publications do not count (or even count negatively!) when it comes to tenure prospects?

How do we account for stability? A string of short-term jobs where you spend much of your time applying for the next opportunity are a different beast than tenure-track or tenured positions! And indeed, even two seemingly stable, tenured jobs can be quite different in terms of teaching load. If your net academic age does not consider the fact that some academics can focus all their energy on research, while others are teaching staff doing as much research on the sides as they can, the aim of “allowing us to make a fair comparison with other applicants” is a hollow promise.

I don’t know, we can also add administration and management duties, a boss who gives you the freedom to undertake whatever research you want (or no boss at all) versus a boss who tells you exactly what to work on (e.g. in a project), the resources you have available like research assistants or money to field work or experiments, if authorship practices in your team differ from common practice in your discipline…

And then we get evaluated by criteria like “excellence” (for grants) or “fit” (for jobs) that are as vague as they are explicit (as DORA requires) — does it really make things better if we adjust for net academic age?

Reviewing activities on a 2-page CV

I’m a big fan of 2-page CV, but in the most recent template I have received from a funder, they also ask to list reviewing activities. On the one hand, I appreciate that they try to acknowledge reviewing activities, on the other hand, I wonder what selection criteria would be appropriate — listing everything would fill most of the 2 pages (and still not tell us much about the quality of the reviews); only listing activities for “prestigious” work kind of defeats the point of trying to acknowledge the less glorious parts of what we do.