I’ve just submitted a review for a potential journal article online. They [the publisher] use one of these systems where there is no direct link to log in (or it’s not set up to do this), and because apparently I didn’t update the password in the password manager last time I changed it, I had to reset it. Easy enough. My first password was rejected because it failed their password rules (compare: https://xkcd.com/936/ and insert your favourite qualifier for these rules). Next I have to “update” my personal details, no skipping allowed. Yes, it’s absolutely necessary for them to know my city! Did they forget that I’m offering my time for this?
Image credit: cc-by davitydave
I’m sure I’m not the first to notice, but it seems to me that peer-review encourages p-hacking. Try this: (1) pre-register your analysis of a regression analysis before doing the analysis and writing the paper (in your lab notes, or actually on OSF). (2) Do the analysis, and (3) submit. How often do we get recommendations or demands to change the model during the peer-reviewing process? How about controlling for X, should you not do Y, or you should do Z, etc.
Unless we’re looking at a pre-registered report, we’re being asked to change the model. Typically we don’t know whether these suggestions are based on theory or the empirical results. In the former case, we should probably do a new pre-registration and redo the analysis. Sometimes we catch important things like post-treatment bias… In the latter case, simply resist?
And as reviewers, we should probably be conscious of this (in addition to the additional work we’re asking authors to do, because we know that at this stage authors will typically do anything to get the paper accepted).
Photo credit: CC-by GotCredit – https://flic.kr/p/Sc7Dmi
It’s common practice to respond to reviewer comments with the phrase “The reviewer raises an important point”, I’ve seen it recommended on many occasions. Today, I had this following gem:
Me (aka reviewer 1): Major point ….
Response: The reviewer raises an important point …
Didn’t I just say so?
Funny thing: Just as I’m doing some light reading on metrics, impact factor gaming, and predatory journals with no real peer-review, I get a revised article to review where there were six (!) reviewers solicitated, all of whom made substantial recommendations.