Tag: DORA

Posted on

Living with DORA

Here’s a blog post by Boris Barbour about living with DORA. I guess we’re seeing some of the ideas in practice, but is it really going to be better?

Less frequent evaluation — I guess this makes sense for everyone who’s currently undergoing annual evaluations for the sake of it. Evaluations when you need them seems logical. But then this means we’re moving even more to an in/out system: once you’re in, you’re in, and you’re free to do whatever. Didn’t get the grant, well you’ll have to live with a smaller research team or more teaching. What about those who are not “in”, who don’t have a tenured position or a clear perspective (a.k.a. “track”) of obtaining one? You’re going to be “out” no matter.

What’s your biggest achievement? — The Swiss National Science Foundation is using this, asking for the three biggest achievements. I’m confused about this. In the context of a job application, isn’t that the cover letter, where you’d highlight just that? Again, this seems fine for those who are “in”. Not getting any publications out, then you can highlight your influence on public policy or something like that. It’s a “pick your own” evaluation thing. What about those who are not “in”? You’re going to try and second guess which of your achievements the evaluators may find relevant. You have all the incentives to inflate your achievements, and none to be humble about what you contributed to the team. If we consider fraud to get publications out, why not consider it here?

Looking at how they write about their own research on social media can indeed be personal and revealing, but if we know that this is (possibly) evaluated, we’re changing the incentive structure.

An interesting thought I found was the idea to ask letters of recommendations from those we mentored and supervised in the past. This is the only idea where the assessment of “quality” does not rely on the candidate’s ability to “brag” and present their achievements in a particular light, or the evaluator being intrinsically familiar with the work and thus able to assess the “quality” directly.

Posted on

DORA … are we getting there?

Yes, we probably all agree that we should evaluate research quality and not quantity. DORA works in that direction, but it avoids specifying what quality means. Perhaps we can even trust each other to identify ‘quality’ and ‘excellence’ just like that.

But consider the following guidelines:

  • “The total number of publications or the number of publications per year is not considered to be the only indicator of performance.”
  • “Each applicant may list up to 10 scientific publications.”

Both of these are attempts to put DORA into practice. In the former, the number of publications per year cannot be used as the sole indicator (“not the only indicator”). In the latter, we actually remove the possibility to do the former (unless we’re evaluating researchers with fewer than 10 outputs).

I don’t know… I’m not convinced we’re changing much other than how we structure CVs and what we highlight. And thinking about it prospectively (early career researchers; planning what research to focus on), can we even guess which research (output) will have a “big” impact on other researchers or society?

Posted on

Academic age

In principle, academic age is simple: the number of years since you completed the PhD. The idea is that your physical age is irrelevant when we evaluate your track records (publications, citations). Instead, we consider how much time you had to build that track.

So far, I guess we’re all on board. However, just looking at the time since your PhD (is that the viva or the graduation ceremony, by the way?) is inadequate if you could not actually do research, like if you were working in industry or engaged in clinical activities (i.e. non-academic employment) or were on parental leave. We can account for this, like the new SNSF CV do, in the spirit of the DORA declaration:

The net academic age is the amount of time you have actually been able to dedicate to research, after deducting interruptions and non-scientific work.

The SNSF template offers: maternity; paternity, adoption, parental leave [as a separate category]; illness or accident; care duties; public service; continuing education; non-academic employment or clinical activities; part-time work; unemployment; other.

However, the more I think about it, the more complicated it gets. For instance, it is increasingly common for PhD students to do paper dissertations, that is, they write a series of academic journal articles rather than a monograph. In terms of research undertaken, this is equivalent, but in terms of publications and citations, a paper dissertation gives you a head start over a classic monograph that may be published a year or so after graduation.

Parental leave and reduced percentages for care reasons seem pretty straight-forward. But what about those who keep working on their research projects during parental leave, do they get a deduction because technically they were on leave, or not because effectively they worked?

Where does academic employment end (or is scientific work the relevant concept? — the guidelines are vague)? In my case, I do a substantial part of my work consulting on commissioned work for government offices etc. Is this academic employment because it entails research and (basic) scientific methods, is this academic employment because the job is at a university (but what if I did the same work in industry, like many of our competitors?), or is this non-academic employment because we produce grey literature (reports that may or may not be published, no peer-review), or because we do not choose our own research question, or because these publications do not count (or even count negatively!) when it comes to tenure prospects?

How do we account for stability? A string of short-term jobs where you spend much of your time applying for the next opportunity are a different beast than tenure-track or tenured positions! And indeed, even two seemingly stable, tenured jobs can be quite different in terms of teaching load. If your net academic age does not consider the fact that some academics can focus all their energy on research, while others are teaching staff doing as much research on the sides as they can, the aim of “allowing us to make a fair comparison with other applicants” is a hollow promise.

I don’t know, we can also add administration and management duties, a boss who gives you the freedom to undertake whatever research you want (or no boss at all) versus a boss who tells you exactly what to work on (e.g. in a project), the resources you have available like research assistants or money to field work or experiments, if authorship practices in your team differ from common practice in your discipline…

And then we get evaluated by criteria like “excellence” (for grants) or “fit” (for jobs) that are as vague as they are explicit (as DORA requires) — does it really make things better if we adjust for net academic age?