Learning how to be a good reviewer

Considering how important reviewing is to research science, I am really surprised by how little thought goes in to teaching students how to do a good job as a referee. I got literally no training--I started getting review requests as a postdoc, and learned on the fly. I am sure I was unreasonably harsh the first few times, and I am truly grateful to the journals that send on all the comments so I could scale my comments with those of other reviewers.

Students are notoriously harsh when it comes to reviewing the work of others. I see it in class, when students give absurdly low marks when asked to score each others' presentations, in journal club when students harsh on the work of other groups, and sometimes in reviews, when I have senior students review papers with me so they can see how I do it. At the same time, students are very sensitive to harsh remarks sent on their own work, but don't have the experience to put either their own reports or the reports they receive in context.

Writing a good referee report is a skill that can/must be learned like any other. Even considering how to phrase something is important if you care about how the comment will be received. "There are a number of typos and grammatical errors that should be double checked before final publication" and "The writer is obviously not a native English speaker and should get someone to edit their horrible writing" both comment on the same thing, but one is more likely to be received constructively than the other.

When I am going over reviewing with a student, I go through how I approach a manuscript with them, then have them write a "review", then go through the review with them before editing into the final report. I never send a student report without doing a review myself, just to make sure I agree with the comments (which will have my name on them). So how do I go about reviewing a manuscript?

1. I have a new file open to type my comments as I go. I don't like to have to try to remember what I was thinking, particularly for long manuscripts.

2. I start with a summary of the manuscript (which I add to as I read). This tells the authors what I think the manuscript is about after reading it. If my summary does not match what they thought they wrote, they will know there is a miscommunication that should be cleaned up. This has actually happened to me once when I was a postdoc--the summary did not reflect what I thought the paper was about. I've also received some great insights into my own work this way, particularly for my first first author paper.

3. I point out glaring typos/grammatical issues until I get past 5 or 6, and then I just add "There are a number of typos and grammatical errors that should be double checked before final publication". I really dislike editing via referee report. If the writing is of a quality that I cannot understand what the authors did (this happened once), I stop reviewing and send it back with a comment to that effect.

4. If the paper is in my area, I sometimes have some suggestions of papers to add to the background (I only sometimes suggest papers from my group. More often, it is something else I think they may have overlooked). If the paper is not exactly in my area, I take more time with the introduction/background, and make suggestions if there is anything I need to look up myself in order to understand the manuscript.

5. I go through each reported experimental result, look at the figure(s) or table, and think about how I would interpret that data. If I conclude something different from the authors, I mention it. If I think the authors are overinterpreting, underinterpreting, or missing a control, I mention that too. I also mention if I agree/if the discussion is appropriate. If any questions come up, I write them down. If the questions are just for interest and are likely beyond the scope of the paper, I preface the question with  "This may be beyond the scope of the paper, but..." to make sure the question is taken in the spirit offered (and doesn't block publication of an otherwise fine manuscript because the editor insists that everything be addressed).

6. I go through the methods section and try to imagine replicating the work from just the methods section.  I often have questions about details here that I think should be reported.

7. I read through the concluding remarks and make sure they make sense and place the work in the perspective of the field.

When I am done, I look through the paper again quickly to see if I missed anything. My final report is usually 1-2 pages long. I've never recommended acceptance without revision, though I rarely reject manuscripts outright (in my opinion, suggesting major revisions and/or a different journal is not outright rejection). It takes me a while to do a good review, but I really appreciate thorough reviews of my own work, so it is worth it to me. I'd say I take a good 1-3 hours to do a typical review (depends on length and my familiarity with the area).

When I am reviewing with a student, usually their first draft is way too harsh, asks for a huge number of additional control or scope experiments, does not differentiate between simple experiments one can expect anyone in the field to be able to do and super-heroic experiments that very few groups can even attempt let alone get usable data from, and is either way too brief or super long. Student reviews are often about showing off a students' knowledge or protecting against looking stupid rather than an attempt to improve the work that was submitted. It is a big mental shift for a student to start thinking of themselves as a knowledgeable scientist rather than as someone who will be graded on their review. Through it takes longer to do a review with a student, I'd rather have my students learn how to do this with guidance then get thrown into the deep end right away.