Tuesday, July 18, 2017

Learning how to be a good reviewer

Considering how important reviewing is to research science, I am really surprised by how little thought goes in to teaching students how to do a good job as a referee. I got literally no training--I started getting review requests as a postdoc, and learned on the fly. I am sure I was unreasonably harsh the first few times, and I am truly grateful to the journals that send on all the comments so I could scale my comments with those of other reviewers.

Students are notoriously harsh when it comes to reviewing the work of others. I see it in class, when students give absurdly low marks when asked to score each others' presentations, in journal club when students harsh on the work of other groups, and sometimes in reviews, when I have senior students review papers with me so they can see how I do it. At the same time, students are very sensitive to harsh remarks sent on their own work, but don't have the experience to put either their own reports or the reports they receive in context.

Writing a good referee report is a skill that can/must be learned like any other. Even considering how to phrase something is important if you care about how the comment will be received. "There are a number of typos and grammatical errors that should be double checked before final publication" and "The writer is obviously not a native English speaker and should get someone to edit their horrible writing" both comment on the same thing, but one is more likely to be received constructively than the other.

When I am going over reviewing with a student, I go through how I approach a manuscript with them, then have them write a "review", then go through the review with them before editing into the final report. I never send a student report without doing a review myself, just to make sure I agree with the comments (which will have my name on them). So how do I go about reviewing a manuscript?

1. I have a new file open to type my comments as I go. I don't like to have to try to remember what I was thinking, particularly for long manuscripts.

2. I start with a summary of the manuscript (which I add to as I read). This tells the authors what I think the manuscript is about after reading it. If my summary does not match what they thought they wrote, they will know there is a miscommunication that should be cleaned up. This has actually happened to me once when I was a postdoc--the summary did not reflect what I thought the paper was about. I've also received some great insights into my own work this way, particularly for my first first author paper.

3. I point out glaring typos/grammatical issues until I get past 5 or 6, and then I just add "There are a number of typos and grammatical errors that should be double checked before final publication". I really dislike editing via referee report. If the writing is of a quality that I cannot understand what the authors did (this happened once), I stop reviewing and send it back with a comment to that effect.

4. If the paper is in my area, I sometimes have some suggestions of papers to add to the background (I only sometimes suggest papers from my group. More often, it is something else I think they may have overlooked). If the paper is not exactly in my area, I take more time with the introduction/background, and make suggestions if there is anything I need to look up myself in order to understand the manuscript.

5. I go through each reported experimental result, look at the figure(s) or table, and think about how I would interpret that data. If I conclude something different from the authors, I mention it. If I think the authors are overinterpreting, underinterpreting, or missing a control, I mention that too. I also mention if I agree/if the discussion is appropriate. If any questions come up, I write them down. If the questions are just for interest and are likely beyond the scope of the paper, I preface the question with  "This may be beyond the scope of the paper, but..." to make sure the question is taken in the spirit offered (and doesn't block publication of an otherwise fine manuscript because the editor insists that everything be addressed).

6. I go through the methods section and try to imagine replicating the work from just the methods section.  I often have questions about details here that I think should be reported.

7. I read through the concluding remarks and make sure they make sense and place the work in the perspective of the field.

When I am done, I look through the paper again quickly to see if I missed anything. My final report is usually 1-2 pages long. I've never recommended acceptance without revision, though I rarely reject manuscripts outright (in my opinion, suggesting major revisions and/or a different journal is not outright rejection). It takes me a while to do a good review, but I really appreciate thorough reviews of my own work, so it is worth it to me. I'd say I take a good 1-3 hours to do a typical review (depends on length and my familiarity with the area).

When I am reviewing with a student, usually their first draft is way too harsh, asks for a huge number of additional control or scope experiments, does not differentiate between simple experiments one can expect anyone in the field to be able to do and super-heroic experiments that very few groups can even attempt let alone get usable data from, and is either way too brief or super long. Student reviews are often about showing off a students' knowledge or protecting against looking stupid rather than an attempt to improve the work that was submitted. It is a big mental shift for a student to start thinking of themselves as a knowledgeable scientist rather than as someone who will be graded on their review. Through it takes longer to do a review with a student, I'd rather have my students learn how to do this with guidance then get thrown into the deep end right away.


Grumpy said...

Great post!

Reviews take me too long, more like 3-5 hours. Somehow I feel obligated to be gatekeeper and search hard for inconsistencies, ask questions about technical stuff I didn't understand, etc.

Grant proposals often take me even longer, sometimes up to a whole day (minus meetings).

My solution is usually to only accept one review (of anything) at a time. But inevitably that means I end up turning down many interesting papers in my sub-fields. seems like it would better for everyone if I just figure out how to review faster...

prodigal academic said...

Hi Grumpy! I usually take closer to 3 hours than 1 hour to do a review (unless it is a communication directly in my research area), but it took me a while to stop considering my role to be a gatekeeper vs someone who is there to check the quality/thoroughness of the work. If I believe the experiments were done properly (from the methods), and I can understand the reasoning behind the discussion and conclusions, I try to stick to comments aimed at improving what is in front of me, rather than looking for potential problems like I do for a proposal. Proposals take me a long time too! :-)

Rheophile said...

I'm with Grumpy - I take way longer than 3 hours, probably closer to 5-6 hrs, unless the paper is fairly short, decent, and in my area.

My process sounds a lot like Prodigal Academic's, especially steps 5 and 6 - I try to interpret what they're doing, model it in my head, and see how I'd reproduce it. Sometimes this points out controls they should have done, etc. I find this is a really slow process, though - I have to sit down, think about their work, make sure I've interpreted everything right, etc. It requires a good amount of intense brain work - much more than just doing minor edits on a manuscript.

I'm reluctant to cut back on the amount of time I spend per manuscript, though, because I've found, in about 20% of papers, major technical errors that invalidated one or more figures or conclusions, and these are not things that I usually find early in the process. Usually, it takes a lot of me going, "wait, why do they see this? is this forbidden by some basic principle?" or a lot of in-depth mental models of their system to spot this.

The one thing I might add to PA's list, which can sometimes frame my thinking, is that if it's a review for a highly-selective journal, I look at the abstract and go, "If they prove everything they claim in the abstract to my satisfaction, is that enough for this journal?" That can help prevent me from engaging in "review creep" where I want more and more and more....