Back to all Advice

How to review a paper 

Created April 16, 2021

Papers published in academic journals are peer-reviewed before being published. You may already have received such reviews on a paper that you submitted for publication. For this system to work, reviewers are needed, and it is ideal if you perform a few such reviews while you are a student. Ideally, you can do it after publishing one or more papers, but before you graduate. There are several benefits to doing this:

• You provide a valuable service to the academic community by using your expertise to evaluate a potential contribution.

• By critically examining someone else’s paper and identifying its good and bad features, you learn how to make your papers better.

• You will hopefully learn from the paper and advance your knowledge in your research area.

Reviewing advice

Here are a few things to keep in mind when performing your review:

• You don’t need to feel completely alone when performing your first review or two. If you are unsure about a particular point, discuss it with your advisor or a mentor. It takes some calibration to figure out what to write in a review.

• Your job is to give the editor advice, not make the final decision about whether the paper should be accepted. The most useful thing you can do is point out the manuscript’s key strengths and weaknesses and critical issues that the authors should address. If there are parts of the paper that you do not feel qualified to evaluate, you can say that in the review. Calling out an area that you didn’t consider can be useful, as then the editor won’t take your silence on a section as an indication that it is problem-free. There will be one or more additional people writing reviews, so the editor will hear from others before making a decision.

• In my mind, a paper is publishable if it makes a new contribution to the field, and thus advances our knowledge. The paper does not need to solve all remaining questions in the field, but it should document the specific problem being addressed, the approach being used, and the basis for the conclusions. When reviewing a paper, I ask, “If this was a published paper in the journal, and I read it, would I feel I learned something of value?” If you would be happy to see it as a reader, then it is publishable. If you would be disappointed to see it as a reader, then it shouldn’t be accepted.

• Be cautious about recommending wholesale changes in scope to the paper. It is your job to decide whether the paper is a useful contribution – not to tell the authors what type of paper you would have written. An exception is when the conclusions are not substantiated by analysis, and the conclusions or supporting analysis need to be changed.

• Be mindful of your tone. Your comments will be read by real people who worked hard on the manuscript, so be respectful and constructive. Even if you recommend rejecting the paper, try to compliment something and leave some advice. Especially for junior authors, encouraging feedback can mean a lot and help them improve (which should be the ultimate goal of this process).

• It is not your job to correct spelling or grammar in the paper. If the writing is abysmal, you should note this as a comment, but you don’t need to point out each specific error. It is helpful if you happen to list a few typos, but don’t spend time looking for all of them.

• Finish your review by the requested deadline. Slow reviewers disrupt the whole publication system, so don’t contribute to the problem. In terms of budgeting your time, try not to use more than a half-day to do the review. It might take you longer than that the first time or two. Like with most of your tasks as a researcher, you could spend more time, but you need to make sure that this service does not disrupt your important job of producing research.

What to include in a review

The format of your review should be as follows:

• A short paragraph explaining the paper’s contribution, some key features of the paper (good and bad), and your evaluation.

• A numbered list of specific items that the authors should address. If you recommend rejecting the paper, list a few significant flaws that led to this decision. If you don’t recommend rejection, this is a list of items that should be addressed before acceptance. The number of comments depends on the paper, but I often list somewhere between five and 15 items in my reviews. If you list more than 15 items, you may be worrying about trivial things, or the paper may be so flawed that it should just be rejected. Number the list so that the editor and authors can refer to them by number in future discussions of these items.

• Each item should be specific so that the authors can identify and understand the issue, and the editor can determine whether a revised manuscript has addressed it. An example of a good comment is:

  “In figure 6, the authors conclude that Z is dependent on X. But that dependence may be an artifact of ignoring Y, which is the more fundamental driver of Z. Smith and Jones (2008) discussed this issue. The authors should address this possibility.”

  Comments should not be hostile or personal. Making vague claims is also problematic. A common but nasty comment is, “The authors have not cited key literature on this topic.” If that is true, you should list some example citations and state where in the paper they are relevant. Otherwise, the author will have a tough time figuring out what you are alluding to. Vague comments make it hard for the authors to understand your concern and respond, and they make it more likely that the revised manuscript will not resolve your concern.

• Your list of comments will likely include a few major or fundamental concerns and several more minor concerns. In such cases, it can be useful to separate the list into two sections, with headings of “Major comments” and “Minor comments,” or to put the most critical items at the top of the list. You could also make a statement in the opening paragraph such as “Items 1 and 2 in the list below are particularly critical for the authors to consider, as without resolution they undermine the entire paper.”

Example paper review

The following example review illustrates the format described above.

[Paper title]

This manuscript presents a study of tall buildings to evaluate how the variation in each structure’s response can be explained by several ground motion properties, particularly those related to the presence of velocity pulses. Velocity pulses are of concern for designers of tall buildings located close to faults, so the problem is one of practical importance. In general, the paper presents novel results. It is suitable for publication, though the comments raise some issues regarding the paper’s clarity and should be addressed before the manuscript is accepted. Items 1 and 2 are particularly important to resolve, as they limit the generality and practicality of the findings.

1. The IMs selected by the authors as good predictors (PGD, VPA, and T_p) are not IMs for which target values could be easily obtained if a tall building designer wanted to utilize these results to select ground motions. Do the authors have any thoughts on resolving that challenge or how the results might be productively used in other ways? Some discussion of this issue is needed.

2. Current design practice uses response spectra as predictors of structural response. However, this manuscript devotes little attention or discussion to elastic spectral ordinates as candidate IMs. Some justification of this should be provided, and ideally, some spectral-acceleration results provided to help benchmark the results in the paper.

3. Page 2, the authors present a short discussion related to “selecting ground motions based on Sa(T1).” Do they mean “scaling motions based on Sa(T1)?” There is no indication in the paper that Sa(T1) was used as a selection criterion, but perhaps I misunderstand what was done.

4. In Section 5, it is hard to get a sense of the relative effectiveness of the scalar and vector IMs, when the former are evaluated using correlation coefficients and the latter using R² values. Putting all of the results in terms of R² values would be more logical.

5. Figure 6 can be omitted. It is difficult to read with all of the superimposed data sets, and there is little discussion of the figure in the paper, indicating that it isn’t a critical figure.

6. In the conclusions, and elsewhere where the topic comes up, the authors should say that an IM is the best of the “considered” IMs (not necessarily the best of all possible IMs). The two statements are not equivalent, unless the authors have additional information not included here that other IMs are all less effective.

7. The last paragraph of the paper brings up the new topic of validation of simulated ground motions. First, it is preferable to avoid introducing a new topic in a Summary section, so this discussion belongs elsewhere in the paper. Second, it is unclear what the authors are claiming. That the results of this paper make validation possible? That this work already provides some validation? If the authors wish to raise this topic, the claim should be reworded to clarify.

My worst ever paper review

The following is the worst paper review I’ve ever received, in the sense of being non-useful. Count how many of the above advice items were violated in this review!

 

The title seems to be somewhat inconsistent (or at least a gross generalization) of the contents of the paper. Please modify the title. 

The Abstract should be written in a way that a general reader can comprehend the message without having to read the paper. As such, the last 2 sentences of the abstract need to be rewritten. 

The introduction is written rather poorly. There should be a stronger and expanded background, a clear problem statement, and clear study objective. 

The stated contribution of the study is vague and rather loosely written. Authors must address this. 

The language of the paper needs to be very clear for the intended audience and users of the methodology. At certain areas of the manuscript, the authors should eschew needlessly esoteric language and embrace clarity of language. 

Authors should review some more relevant literature on the subject.