Instructions for Reviewers

9 minute read

ACL-IJCNLP 2021 Instructions for Reviewers

Thank you for agreeing to help in the reviewing process for ACL-IJCNLP 2021! In order to ensure that all valid submissions receive a fair and thorough review, we would like to bring your attention to a few guidelines for reviewing papers. Please read these instructions before you start reviewing papers. All reviewers are assigned no more than 6 reviews each. Since we have a joint long and short paper deadline, please be mindful that you may have a combination of long and short papers to review.

Important Dates

Your reviews are due on Saturday, March 20, 2021 (11:59pm anywhere on Earth). The rebuttal period starts on March 25, thus it is crucial that you complete your assignments by March 20.

Please note that there is a reviewer discussion period from April 1 to 7 after the author response. Your duties are marked in bold. Don’t leave reviewing to the last minute!

Review Period February 25, 2021 - March 20, 2021
Author Response Period March 25, 2021 - March 31, 2021
Reviewer Discussion Period (ACs lead discussion) April 1, 2021 - April 7, 2021

Confidentiality

Please note that the content of any submission to ACL-IJCNLP 2021 and the content of discussion about submissions, are confidential.

The Review Form

We adapt the review forms and the instructions from ACL 2020, EMNLP 2020 and NAACL 2021. The review form consists of four main sections.

1. In-Depth Review. This section is for you to give your overall assessment of the paper and to provide evidence to support your opinions. There are ten subsections:

  • The core review: This is the most important part. It should include your view of the main contributions that the paper intends to make and how well it succeeds at making these contributions. From your point of view, what are the significant strong and weak parts of the paper and the work it describes? This could be a 2 paragraph (or longer) essay and/or a bullet list. Remember to describe how the work advances the state of knowledge in computational linguistics and/or highlights why it fails to make a sufficient contribution.

  • Reasons to accept: Please briefly summarize from your core review the main reasons why this paper should be accepted for the conference, and how the ACL community would benefit from it. You may refer back to your review to provide more context and details.

  • Reasons to reject: Please briefly summarize the main reasons that this paper cannot be published and presented in its current form. What are the parts that would need to be improved in order to advance the state of knowledge?

  • Reproducibility: Following EMNLP 2020 and NAACL 2021, we provide a reproducibility checklist in an effort to increase reproducibility of the research work in NLP. In the review form. Please answer the following two questions.
    • “How do you rate the paper’s reproducibility? Will members of the ACL community be able to reproduce or verify the results in this paper?” Scores of 1-5 are used to assess this aspect. The detailed explanation for each point level is provided in the review form. N/A may be used for the papers that do not include empirical results.
    • “Are the authors’ answers to the reproducibility checklist useful for evaluating the submission?”. Three choices are provided for this question (Very useful, Somewhat useful, and Not useful). Note that this question is for us to collect feedback regarding the usefulness of the reproducibility checklist, and is not about evaluating the paper itself.
  • Ethic review: This year, authors are allowed extra space after the 8th page (or the 4th page for short papers) for a broader impact statement or other discussion of ethics. Regardless of whether the paper includes an impact statement, you need to read the whole paper carefully and determine whether the paper has some potential ethical issues. If that is the case, you should flag the paper by choosing “yes” to the “ethical concerns” question. The flagged papers will then be reviewed by an Ethics Advisory Committee (EAC).
    • Please read the FAQ for ethical discussion standards and the questions in the ethics section of the review form for more guidance on some problems to look out for and key concerns to consider relative to the code of ethics.
    • If you flag a paper, it is very important that you provide justification as detailed as possible to help EAC to understand your concern.
    • Note that more than 30% of the submissions include ethics/impact statements, and most of them are unlikely to have ethical issues. Therefore, please read the statements and use your best judgment to determine whether the paper should be flagged. Please don’t automatically flag a paper simply because it includes an ethics/impact statement.
    • Regardless of whether you flag a paper, you should review the paper carefully and answer the rest of the questions on the review form.

  • Anonymity requirement: Please indicate whether you know the identity of some authors of this paper. If yes, please provide more details (e.g., the url of the version posted online) in the “Author identify justification” box.

  • Overall recommendation: Here you are asked to synthesize the above and come up with your own recommendation for the paper.
    • Like ACL 2020, we have used a 5-point scale with a half point increments. The detailed explanation for each point level is provided in the review form. These numbers are just a concise way of expressing your overall opinion and relative importance of the factors mentioned above.
    • Different from ACL 2020, we are allowing a rating of 3 (ambivalent). Please try to take a stand on whether the paper is above or below the borderline, e.g., by selecting 2.5 or 3.5. However, if you think this is indeed a borderline paper or you are not able to decide, you should use 3.
    • Decisions will be made not just on the scores and certainly not on average scores, but will also take into account the whole review, reviewer discussion and area chair meta-reviews and recommendations. However, it is important to align your recommendation with the reasoning given above, so that authors will be able to understand the motivation for the recommendations and how decisions were arrived at.

  • Reviewer confidence: This section should be used to inform the committee and authors how confident you are about your recommendation, taking into account your own expertise and familiarity with this area and the paper’s contents.

  • Recommendations for awards: For Best Paper Award, please be open minded and feel free to nominate good quality papers even though they may not be the typical types. These can be a survey paper, an opinion paper, a paper about resources and datasets, a paper for low resource language, an analysis paper, etc. A committee will evaluate best paper candidates, and we would like to have a wide variety of paper types in the candidate pool, not just vanilla empirical research papers.
  • Paper focus (for Theme paper only): If the paper is a theme paper, please check all the topic(s) covered by the paper.

2. Questions and additional feedback for the authors: Since we will have an author response process, for questions you would like the author(s) to respond to during the response period, please include them here. This is also the place for you to give suggestions to the authors to help them improve the paper for the final version (or a future submission).

3. Changes after the rebuttal period: There will be an author response period. It is important for you to check whether author responses have cleared up your questions. This may influence your overall recommendation and the core review. If that’s the case, please update your recommendation and review in Sections 1 and 2 accordingly, and briefly explain your decision for changing or keeping your original overall recommendation in Section 3.

4. Confidential information to the committee and chairs: Your comments in this section will not be shared with the authors. There are two boxes in this section. The first one will be shared with your peer reviewers and the area chair, and the second one will be shared with the area chair, senior area chairs, and program chairs.

Supplementary Materials

Supplementary materials are allowed as a stand-alone document uploaded as additional files. Supplementary materials are, as the name suggests, supplementary, and you have no obligation to read them. You should treat them like other citations in submissions that may be helpful in understanding background or details beyond the scope of the paper itself.

However, as noted above, given the requirement for reproducibility, authors may provide additional information about their datasets and experiments in Appendix, and attach a zip file with resources such as code and data. Please take some time to check those, if applicable.

Secondary Reviewers

As in most previous NLP conferences, you are allowed to solicit help from others. However, when it comes to writing the final review and giving the final scores, we expect you to take the secondary reviewer’s review, revise it and adjust the scores when you see fit. Essentially, the final review should reflect your own opinions about the paper, and you need to be able to justify the opinions you present in the final review.

When you send the secondary reviewer any material related to the submission (e.g., the paper itself, the supplementary materials), please remind him/her to keep the material confidential.

Format of Submissions

The program chairs, the senior area chairs, and the area chairs have identified some submissions that violate our formatting guidelines and have desk-rejected those submissions. However, it is possible that some violations were not detected in the first pass. Therefore, if you think the paper violates the format guidelines, please mention that on the review form, and highlight that in Section 4.

General Guidelines for ACL Reviewing

Please take a balanced approach when reviewing the papers. On the one hand, we would like to have a solid technical program with high-quality papers describing a complete piece of work. On the other hand, we also want a broad and interesting program, so please keep an open mind when evaluating and recommending the papers assigned to you.

Please note that a short paper is not a shortened long paper. Instead short papers should have a point that can be made in a few pages and present a focused contribution.

Following ACL 2020, we suggest that you consult some of the excellent advice from experienced reviewers and conference organizers on the web. In particular:

Reviewer Mentoring Program

  • This year, we have a Reviewer Mentoring Program. The program is organized by the mentoring committee:
  • The committee has made videos specifically for ACL-IJCNLP 2021, and all the reviewers are welcome to watch the videos before starting reviewing. The videos and other information will be available at:
  • For the “first-time reviewers” who have signed up for one-on-one mentoring, the mentoring committee will contact you directly with more information.

If you have any questions about your assignments, you should contact your senior area chairs directly. You should have already received an email from them via the START system.