So You've Been Asked To Review A Paper...

Peer review is an important, unavoidable part of being an academic researcher. We recieve peer review reports on manuscripts we submit, and we’re often asked to write peer review reports for manuscripts written by others. We start receiving reports as soon as we start writing papers – usually during our PhD studies – and we can start fielding review requests at any time, particularly once we’re already in a publisher’s database as an author with a few ‘research topic’ keywords attached to our names.

I don’t know about you, but no one ever really told me how to write a peer review. My first review wasn’t until my first postdoc position, at which point I’d received a fair few reviews myself, so I modelled my first review on the most constructive ones I had received. Since then, I’ve written more and I’ve received more, and during that time I’ve developed some opinions on how best to conduct the peer review process. Recently, I’ve had the pleasure of writing several joint reviews with younger colleagues who have never written a review before, and so I’ve had the opportunity to put some of these thoughts and opinions into writing as a sort of guide to colleagues who are new to the reviewing process. What follows is a modified version of the guide I put together for my colleagues, adapted for use on this blog.

So You’ve Been Asked To Write A Peer Review

At this point in my career I’ve reviewed around 30-40 manuscripts for a variety of journals (mainly the APS Physical Review family), and as many of the same editors keep coming back to me time and time again, I’m forced to assume that either they’re desperate or I’m not terrible at being a reviewer¹. With this in mind, then, here are my tips on how to approach being asked to peer review a manuscript.

One common question that I don’t have an answer to is “How long should a peer review take?”, as this depends on a lot of factors. How long is the paper, how precisely does it overlap with your area of expertise, how experienced are you at writing peer reviews, how well-written is the paper…? The list of variables is endless. As a rule of thumb, I try to read the paper and draft a review over the course of an hour or so, then I put the paper aside for a few days and come back to it with fresh eyes and spend a little while considering my comments and making sure I still agree with myself. There have been many times where I’ve significantly revised my opinion of a paper during this stage, after realising that I’d completely misunderstood some key point, or conversely noticing some major issue that I hadn’t spotted first time. And if I’m honest, I still often spend far too long on reviews, but this is something I think I’m getting better at…!

A Few Things To Keep In Mind

1. Remember we’re all (supposed to be) on the same side: Your job as a reviewer isn’t to play gatekeeper, nor to pass final judgement on whether a paper does or does not belong in a particular journal. At the end of the day, the decision lies with the editor; all you’re doing is advising them. Don’t feel guilty about writing a negative review if the manuscript warrants it, but equally don’t let the illusion of power go to your head. I get the impression that many people – particularly early on in their careers – treat peer review as an adverserial process, both as authors and referees. It’s easy to treat reviewing a paper as some sort of homework excerse, e.g. ‘Find all the flaws in this paper!’, and return a review that looks like an extensive list of errors, but there’s more to a review than just that. The aim here is to make the paper the best it can be, whether it gets into the journal you’re reviewing for or not.

2. Don’t write anything you wouldn’t say to the authors in real life: Snark can be funny, but its place is when chatting with your friends, or on Twitter or Reddit, but absolutely not in a professional context. For example, a friend of mine once wrote “wtf lol” in a draft review, then forgot to remove it from the final version before sending it to the editor - needless to say they were extremely embarassed when they realised. I’ve never (to my knowledge) had anyone successfully figure out that I reviewed their paper, but even when the review process is anonymous I always try to conduct myself as I would in person. A peer review isn’t the place to amuse yourself at the cost of others; you can be witty without being mean, and you can be critical of a piece of science without being disrespectful to the human beings who put a lot of time and effort into getting it to the submission stage.

3. Be constructive in your criticism: There’s nothing wrong with being negative if you can suggest ways to improve the paper, but unless the manuscript is really without any redeeming features, try to be constructive even when requesting major changes. I once received a review that read, in full, “Paper not suitable in style nor substance for Physical Review Letters. In addition, in my view, the results are wrong”. This is an example of a desperately unhelpful review, as there’s nothing constructive in it and nothing I could really act on. A good negative review will explain why it’s negative, and - if possible - will explain what the authors need to do in order to improve the manuscript, giving the authors a list of points they can act on when revising the manuscript. Even if the editor decides to reject the manuscript and it gets sent instead to another journal, being constructive in your comments will help the paper to be the best it can be no matter were it eventually ends up.

4. You wouldn’t be asked if your opinion wasn’t valid: Depending on your personality type, you might find being asked to write a review rather intimidating. It can feel like a lot of responsibility, and you might doubt whether you’re qualified, but remember they asked you for your opinion. Unless the paper is really in a different field – which can happen, as sometimes all the editors have to go on is a list of context-free keywords in a database somewhere to inform them about your area of expertise – or you really think you can’t in good conscience review a manuscript due to some conflict of interest, then it’s worth telling that little insecure voice in your head to take a break and let you get to work. A good editor will take your experience - or lack thereof! - into account, and won’t necessarily take your review at face value.

5. You can say ’no’!: Finally, remember that if you receieve a request for something that’s not in your field, something you really don’t feel qualified to review, a manuscript where you have a conflict of interest, or even just if you don’t have the time or plain don’t want to review a paper right now, you can always say ’no’ without any negative consequences. We’re not paid for our work as reviewers, after all, and ultimately many for-profit journals maintain obscene profit margins off the back of this unpaid labour we perform out of professional courtesy and a sense of duty to the research community that we’re a part of.

How to Structure a Review

I tend to structure a review in the following way:

1. First paragraph: a concise one- or two-sentence summary of the paper. “The manuscript by X does Y, and this is interesting/novel/exciting/strange because…”. This serves both to summarize the paper for the editor, helps me synthesize the ‘big picture’ of the paper into a single sentence and provides an easy way for the authors to tell if I’ve misunderstood the point of their paper.
2. Second paragraph: summarize the positive parts of the paper. I think this is quite important, as it’s rare that a manuscript is ever totally without merit. This also helps reinforce to the editor and author which parts of the paper you value most, which may not be the parts the authors expected or intended, and so even positive comments may still be useful when the authors come to revise the manuscript.
3. Third paragraph: main criticisms and any negative aspects. Typically I’ll have one or two major concerns/criticisms/misunderstandings that I’ll want to bring up here. This is where I include things that I see as barriers to publication, parts of the paper I would like to see fixed or explained more clearly before I’d be comfortable recommending publication. Of course, in wonderful and rare cases, there are no such issues and I have nothing to say here!
4. Fourth paragraph: sum up all of my comments and conclude with my recommendation. If negative, I’ll try to say here what changes I’d like to see that would make me change my opinion.
5. Final part: Itemised list of minor criticisms and corrections. (For example, I don’t go out of my way to find typos and grammatical errors, but if I do find any, then this is where I’ll mentioned them.)

Note that when I say ‘paragraph’ I really mean just one or two sentences - a review doesn’t have to be War And Peace! Sometimes I’ll combine the first and second paragraphs together, or sometimes the second and third, but for a typical review the points usually occur in this order. (This goes both for reviews I’ve written and reviews I’ve received). And if I’m reviewing a revised version of a manuscript that I’ve already seen, I’ll usually write something much shorter as the authors will (hopefully!) have addressed most of the main points and I’ll have less to say.

Confidential Remarks to the Editor

Remember that in addition to the part of the review that the authors will see, you usually have the option to write some brief remarks to the editor that will remain confidential. I often use this to explain my comments, and let the editor know which ones I expect will be easy to address, which I think will be tricky, and to point out elements of my review where I’m not sure of myself and want to give the authors the benefit of the doubt.

Be Prepared to Admit You’re Wrong

It can happen that your brilliant, insightful peer review might hinge on a misunderstanding. It could be that you missed something, that the authors weren’t entirely clear, or something somewhere in between. Or, it may be that you wrote your initial review on a bad day and if you’re sent a revised manuscript back for a second round, you might reconsider your stance and view the manuscript more favorably. Whatever happens if/when you’re sent the manuscript for a second or a third round, approach it with humility and don’t get defensive if it turns out you were wrong. Again, we’re all working towards the same goal, that of making the paper the best it can be regardless of which journal it gets sent to - if you were wrong, put your ego aside and treat it as a learning experience. I’ve learned some very interesting things from authors who patiently pointed out parts of my review where I’ve misunderstood their work, so it’s worth keeping an open mind. Equally, I did once review a manuscript where the results were either fabricated or completely misrepresented, so don’t be shy in calling out things that you don’t understand or can’t reproduce - just do it with the hope that the authors are able to explain to you what they’ve done, and why it’s justified.

Closing Thoughts

Of course, even though I present this as a guide to peer review, it’s really just a loose collection of thoughts and opinions that are worth keeping in mind when writing a review. I don’t stick dogmatically to the above, and often mix things up a little, particularly if the review is unsigned (as most are) and I’m trying to stay anonymous – so don’t assume that anyone using the above structure is necessarily me! If I’m reviewing two papers in a row from the same group, for example, I’ll change the format, and may even switch between US and UK spelling. If you’re perhaps just starting out in academia and want to make sure you stay anonymous for fear of repercussions (although in my experience so far I’ve never heard of this being a real risk), beware of any idiocynrasies in your writing that you may want to eliminate from your ‘reviewer’ voice so that it’s difficult to guess who you are purely from your writing.

So what do you think? Do you agree with the above, or have I missed something? Are you an editor of a scientific journal and want to chip in with your opinion? If so, I’d love to hear it! You can get reach me by e-mail (address available on this site) or on Twitter.