Tips on Reviewing

Shane Bergsma
January 9, 2013

Have you been asked to be a reviewer for a paper or journal and are you now wondering what it involves? I think it's a shame that junior researchers are not given specific training in reviewing. Here are some tips that I prepared for a graduate student. I'm speaking from my perspective as someone in computer science. If you've already reviewed a few times, know how things work, and are just looking for advice on improving your own reviews, please skip to the actual tips below.

I found reviewing hard at first, and it's certainly still a chore that seems to distract me from my own research, but I take pleasure in doing a good job at it. I won the Outstanding Program Committee Member Award at AAAI in 2011.

Being a reviewer has two main benefits to you:

(1) You can put it on your c.v. and it shows you are providing service to the community.

(2) You will learn to evaluate your own research more effectively, and look at your work through the eyes of a reviewer. You will not only start to write better papers, but you might select directions in your research better. [In general, as you write better papers, your research will improve. You can save yourself a lot of time by asking yourself as you're going: "Will what I'm doing now actually be useful to someone? Will I be able to explain what I'm doing?"]

I recommend keeping a folder somewhere for all the reviews and scores that you will complete over the years.

How a submission becomes a publication and where you fit in

Generally, a conference will have one or two Program Chairs, who then recruit a number of Area Chairs for specific research areas. These Area Chairs recruit the reviewers. As a reviewer, you will be either called a 'Reviewer' or a 'Program Committee' member. It's worth recording what your title is as you go; I've come across a fellowship application that asked me to distinguish between the two roles in my past reviewing duties, and it was a real pain to try to figure it out years after the fact. For a journal, you might be contacted by an 'Editor', 'Editor-in-Chief', 'Action Editor', or any number of other titles; I focus mostly on conference reviewing below.

Basically, as far as I can tell, being a PC member is the same as being a 'Reviewer' except you have a fancier title. Every conference seems to be different, but here's the typical workflow:

(1) Papers are submitted to the conference and then get assigned to a particular area. The assignment is mostly based on the keywords you choose when submitting a paper, but program chairs/area chairs can always re-route them if they think they are mis-assigned.

(2) Then usually there is a bidding period within a few days of submission where the reviewers in an Area go through the papers assigned to that Area and pick out which ones they would like to review.

(3) The area chairs then choose which papers to assign to which reviewers (taking into consideration the preferences and possible conflicts of interest). While the number you get assigned depends on the number of submissions, which is hard to predict in advance, I generally find I am assigned about 3 papers at most of the ACL-family conferences that I review for (although you might get 3 long papers and then later 3 short ones).

(4) You could have anywhere from 2-3 weeks to review the papers (e.g. for a workshop paper) to over a month (for major conferences)

(5) The area chairs then rank the papers based on the reviewer scores and select the ones for the final program. Decisions + reviews get sent to the authors. Some conferences (like ACL) have an 'author feedback' period where authors can comment on the reviews and reviewers can fix up any mistakes they may have made before final decisions are made. Conferences run on a tighter schedule will likely not have this. Also, occasionally a diligent Area Chair will themselves write extra reviews in borderline cases.

Anonymity and Review Revising

The authors never see the reviewer names. For most conferences, the reviewers never see the author names (although you know who wrote the paper once it's published), while for journals, you usually do see the author name. Also, I'd say 50% of the time, your identity is also hidden from the other reviewers, otherwise you know who else is reviewing once all the reviews are submitted. You can almost always read the reviews of the other reviewers after you submit your own review and use them to adjust your own scores. I swear that I once saw another reviewer submit a 1-line review with dummy scores, then read the other reviews and submit a more detailed review in line with the other reviews... I don't think that reviewer even read the paper! So don't do that.

The Actual Tips

If the area is in line with your research interests, then your best strategy is to read the paper and ask yourself, 'would it be useful to me to see this paper [if I worked on this particular topic]?' If the paper is well outside your own expertise, you can give yourself a lower score for 'reviewer confidence' in the review form.

For tips on actually completing the review, I would recommend reading the following page that I created when I was teaching a graphical models class at JHU a few years ago:

Critique/Review Form

That's pretty much what the form will look like for most ACL-family conferences. The advice is taken from advice that I have received over the years from venues like IJCAI and ACL.

Actually, it's already easier to review now than when I started out. My first review forms just had the categories and then a score from 1-5 to fill in. All the text that they added around the questions/numbers makes it easier now, in my opinion. I credit Jason Eisner with improving reviewing forms back when he was program chair for EMNLP 2007.

If you're reading this, then you're obviously keen to do a good job at reviewing. Thank you. Area chairs and other reviewers will take note of your hard work. You might even win an award.