FAQ for the Decision Process, Camera-ready Version, and Findings

14 minute read

May 10, 2021

The whole review process is quite complicated, and may be mysterious to many authors. Thus, we want to explain the process a little bit in Section 1 before answering more specific questions.

Section 1: The Review Process

Below are the main steps in the review process, which is very similar to those from previous conferences:

  1. 2/2-2/24: Paper check and AC/reviewer assignment
    1. Papers are assigned to tracks by Program Chairs (PCs) and checked by Senior Area Chairs (SACs) and PCs. Let's call these tracks the technical tracks, in contrast to the EAC track (see Step 3).
    2. Three reviewers and one area chair (AC) are assigned to each paper.
  2. 2/25-3/20: Reviewers review papers
    1. Reviews include two parts: non-confident part (which was sent to authors on March 25), and confident part visible only to ACs/SACs/PCs.
    2. If the reviewers or the AC had ethical concerns about the paper, they would "flag" the paper by clicking "yes" for the ethical question on the review form, and provide a justification for their concern.
  3. 3/21-3/24: EAC check and reviewer assignment
    1. PCs add the flagged papers to a special track for the Ethic Advisory Committee (EAC).
    2. EAC chairs then read the paper to determine whether it needs a full EAC review. If so, one or more EAC reviewers are assigned to the paper, and the EAC reviews would be due in mid-April.
  4. 4/1-4/7: Reviewer/AC discussion period
    1. The AC and reviewers of technical tracks discuss the paper in softconf's discussion board. The reviewers are asked to update their reviews in softconf if they change their opinions or have any new comments to add about the paper based on the discussion.
  5. 4/8-4/14: The AC writes a meta review which has three parts:
    1. AC recommendation, which uses one of four labels: 1-Accept, 2-Possible-Accept, 3-Possible-Reject, 4-Reject.
    2. AC summary of reviews (including his/her own opinion) of the paper, which is visible to authors.
    3. AC's confidential comments to SACs/PCs.
  6. 4/15-4/24: SAC recommendation which also has three parts:
    1. SAC recommendation, which uses one of four labels: 1-Definite-Accept, 2-Possible-Accept-to-Main-Conference, 3-Possible-Accept-to-Findings, 4-Reject.
    2. SACs provide a ranking of papers with the 2-Possible-Accept-to-Main-Conference or 3-Possible-Accept-to-Findings recommendations.
    3. SACs' comments to PCs
  7. 3/25-4/24: EAC recommendation:
    1. EAC reviews are due in Mid-April.
    2. EAC chairs read the EAC reviews and inform PCs of their decisions, which include comments and one of three labels:
      1. E1: The paper does not have serious ethical issues
      2. E2: The paper has some serious ethical issues and thus, if the paper is accepted, the camera-ready version needs to be checked by EAC to ensure ethical concern has been addressed in that version.
      3. E3: The paper should be rejected due to ethical concerns.
  8. 4/25-5/5: PCs make final decisions:
    1. Based on the input from the technical tracks (reviews, discussion board, AC recommendation, SAC recommendation, etc.), PCs use one of three labels for each paper:
      1. P1: Accept-to-Main-Conference
      2. P2: Accept-to-Findings
      3. P3: Reject
    2. For P1 or P2 papers (not P3 papers), EAC chairs write a meta review that highlights the changes that EAC recommends authors to make in the camera-ready version.
    3. For P1 or P2 papers, PCs will use the following four labels for the final decisions:
      1. If EAC recommendation is E1, the paper is accepted and the final label is Accept-P1 or Accept-P2.
      2. If EAC recommendation is E2, the paper is conditionally accepted, and the status is Accept-P1-condition or Accept-P2-condition.
      3. None of the papers get an E3 label from the EAC.
    4. For P3 papers, PCs will mark them as Reject-P3.
    5. PCs send out notifications to all authors.

Now, if you go to softconf, you can see your papers in one of five categories:

  • Accept-P1: accepted to the main conference
  • Accept-P1-condition: conditionally accepted to the main conference
  • Accept-P2: accepted to Findings
  • Accept-P2-condition: conditionally accepted to Findings
  • Reject-P3: reject

In summary, the final decision is based on the input from all the steps mentioned above:

  • The submission itself and the author responses
  • Reviews from 3+ reviewers: including the part visible to authors and confidential comments to ACs/SACs/PCs.
  • AC recommendation: including a recommendation label, AC summary, and confidential comments to SACs/PCs.
  • Reviewers/AC discussion messages in the softconf discussion board.
  • SAC recommendation: including a recommendation label, rank of the papers in the area, and comments to PCs.
  • EAC recommendation: including an EAC label (E1, E2, E3), EAC comments confidential to PCs, one or two EAC reviews, and an EAC meta review for accepted or conditionally accepted papers.

In addition, we target an acceptance rate of the main conference and the Findings to be around 21-24% and 15%, respectively.

Section 2: Questions about the review process

This section includes questions that may be relevant to more than one category.

Q1: How does the review process work?

A1: See Section 1

Q2: The AC recommends Accept. Why is my paper still rejected?

A2: As explained in Section 1, AC recommendation is just one of many factors used in the final decision. For instance, some ACs disagree with reviewers: they may recommend Accept when some reviewers insist on the paper being rejected, or the other way around. PCs also look at other factors such as author rebuttal, three reviews, the discussion in the discussion board, SAC recommendations, EAC recommendations (for flagged papers), and the submission itself.

Q3: My overall recommendation scores are 4, 3.5, 3.5. Why is my paper still rejected (or accepted to Findings, but not to the main conference)?

A3: Just like in Q2, review scores are just one of many factors considered by PCs. For instance, some ACs point out that in some reviews, the overall recommendation scores do not match the reviews (e.g., a largely negative review with a score of 3.5), but reviewers do not want to change their reviews or overall recommendation scores for various reasons, even when prompted by ACs. In that case, PCs read reviews, AC recommendation and SAC recommendation in order to reach a final decision.

Q4: My reviews are largely positive but the paper is rejected due to an EAC review. Does EAC have the one-vote veto right?

A4: No. EAC recommendation is only one piece of information used in the final decision, as explained in Section 1.

Q5: Where can we see revised reviews (or comments to our rebuttal)?

A5: The reviews sent to authors on May 5 are the revised reviews. If reviewers had some comments to the rebuttal that they wanted the authors to see, they would include that in their revised reviews.

Q6: How come the reviews remain the same after we spent so much time writing the rebuttal?

A6: Many reviewers were discussing the papers in the discussion board at softconf, and, after the reviewer/AC discussion, they decided to keep their original reviews because they were not convinced by the authors’ rebuttal. In that case, the new reviews will be identical to the original ones.

Q7: Some reviewers (or the AC) misunderstood our paper. Please reconsider.

A7: If some (but not all) reviewers or the AC misunderstood your paper, the overall recommendation scores tended to vary a lot among reviewers or AC’s meta review would disagree with the three reviews. SACs and PCs paid special attention to such papers, and the final decision is the result of considering all the input including AC and SAC recommendations and the discussion board.

Q8: Can we find out whether the reviewers of my paper had read my rebuttal?

A8: The review form contains a question asking reviewers whether they have read the rebuttal. We are collecting the statistics on that question and can release that statistics later if needed. However, we do not plan to release that information for individual papers for two reasons:

  • Some reviewers read the rebuttal, discussed the paper in the discussion board, but for some reasons did not answer that question in the review form. Thus, the answer for an individual paper might not be very accurate.
  • When PCs created the review form in Jan 2021, this question was marked as visible to ACs/SACs/PCs only. Thus, PCs should not change the visibility setting of the question after the fact.

Q9: I have questions about EAC review or meta review. How can I get some clarification?

A9: Please read the Ethics FAQ first at https://2021.aclweb.org/ethics/Ethics-FAQ/. If you still have questions, contact the EAC chairs at acl-2021-ethics-chairs@googlegroups.com directly.

Q10: After the rebuttal, some overall recommendation scores decreased. Why?

A10: This is likely to be due to the reviewer/AC discussion. Some reviewers’ opinions could change due to the discussion or after reading authors’ rebuttal, so the scores can go up or down in the revised reviews. The change of the scores shows that the reviewers have read the author rebuttal.

Reviewers were asked to revise their reviews to reflect their new opinion of the paper, but that might not always happen. Unfortunately, we cannot make the information in the discussion board available to the authors due to the nature of double-blind reviews.

Q11: The AC raised some issues that were not found in the three reviews, which we could not address in the rebuttal period. We disagree with those points.

A11: ACs can point out issues that were not identified by the reviewers. That is why AC recommendation was one of the factors for the final decision. As for your disagreement with AC’s opinion, please see A7 above.

Q12: My paper is marked as accepted to the main conference (or the Findings), but it includes an EAC review and a EAC meta review. Does that mean my paper is conditionally accepted instead?

A12: If a flagged paper was determined by the EAC that a full EAC review was necessary, the paper would get one or EAC reviews. If such a paper was accepted to the main conference or Findings, an EAC meta review would be added. Thus, having those reviews does not mean that the paper is conditionally accepted. If you want PCs to double check for you, please email PCs with your submission ID.

Q13: Some reviewers raised new issues in the revised reviews which we have no chance to respond to. Had they mentioned that in their original reviews, we could have addressed their concern.

A13: It is not uncommon for new issues to be raised during the reviewers/AC discussion period, and reviewers can include such issues in their revised reviews. Given the short review cycle, it is impossible to have another round of author rebuttal after the discussion period. Nevertheless, as mentioned in Section 1, reviews are just one of many factors considered in the final decision.

Q14: Now that my paper has been accepted to the Findings, can I post it to arXiv.org now?

A14: Yes, that should be fine.

Q15: For some reasons I have not received an accept/reject notification from you. What’s the status of my paper?

A15: Please go to the softconf site, click on your paper, you should be able to see the status and reviews of your paper.

Section 3. Questions about the camera-ready version

Q16: Can I make minor changes to the title in the camera-ready version?

A16: Yes, minor changes to the title are allowed.

Q17: Can I make any changes to the author list in the camera-ready version?

A17: No, you cannot make any changes to the author list. The authors’ information was used for COI (conflict-of-interest) detection, and COI is a factor that affects many stages in the review process (e.g., AC/reviewer assignment). Thus, no changes to the author list is allowed in the camera-ready version. In fact, in the final submission form, the author field is not editable.

Q18: Where can we find the templates for the camera-ready version?

A18: We are working with publication chairs on this, and will contact authors above this once the templates are available.

Q19: I don’t see the final submission page. Why?

A19: The Final Submission Page will be open by May 14, 2021 for authors to start uploading their camera-ready version. Before that deadline, the authors will NOT be able to see the camera-ready submission link at softconf.

Q20: What is the page limit for the camera-ready version of the paper?

A20: You will have one extra page for that, 9 pages for long, and 5 pages for short papers, plus unlimited space for references and the impact statement.

Q21: Should the appendix be part of the main paper? What’s the page limit for the appendix? What style file should I use for the appendix?

A21: We are checking with our publication chairs on this. Stay tuned.

(We leave some space for Q22-Q25 here, which will be added once we hear back from publication chairs)

Section 4. Questions for Findings

This section is modified from the Finding FAQ from EMNLP 2020 at https://2020.emnlp.org/blog/2020-04-19-findings-of-emnlp

Q26: What criteria are used to determine whether a paper should be accepted to the main conference or Findings?

A26: The low acceptance rate for the ACL-IJCNLP 2021 main conference means many high-quality work would have to be rejected without the Findings category. That’s the main motivation for us to have this category. Thus, Finding papers are ones with solid work but cannot make it to the main conference due to the low acceptance rate.

Q27: Does Finding have its own ISBN or DOI?

A27: We expect that Findings will grow with future conferences (e.g., having a new volume associated with each conference), and become a recognised and respected publication venue within the field. Initially, we will not be seeking to have Findings indexed in Scopus and Web-of-Science etc. - this is a complex process, and TACL is only now applying after 7 years of operation - but this is a natural option we might consider in future years.

Q28: Why might you want your paper to appear in Findings?

A28: Those with papers accepted to Findings will have the option of having the work accepted largely as-is (with the exception of Accept-P2-condition papers), and being made available online promptly. This will be an attractive option for work that is particularly time-critical, e.g., the novelty of the work would be diminished if publication were delayed by being rejected from ACL. This will also help for papers caught by overlapping anonymity periods between conferences, which can preclude public release for a long period. In this case, Findings can offer the option of a quick publication. Note that authors whose papers are accepted for publication in Findings can, of course, opt out, by withdrawing or simply not submitting a camera-ready copy of the paper.

Many papers languish on arXiv without ever being published in a peer-reviewed venue because they were overtaken by the accelerating pace of NLP research. This new publication option will give more authors the chance to publish papers that are good and decent contributions at a given point in time, even if their long-term impact may be modest. This will also reduce the amount of unreviewed work in our field, and thereby increase quality.

Q29: Is there any advantage of publishing in Findings versus an ACL workshop?

A29: This depends on the publication prestige, which we would expect to be higher for Findings than for most workshops. Moreover, workshops increasingly tend to move to non-archival format.

Q30: Why should the authors agree to publish in Findings if they know the paper was only narrowly rejected from ACL, and so might get into another conference?

A30: This is a judgement call to be made by the authors. Some authors may want to take the acceptance, rather than risk another rejection. This might be more pertinent if the paper has already been rejected from another publication venue, or if the author needs to fill out their CV, e.g., a PhD student wanting to secure a postdoc or faculty position. Other reasons might be that the importance of the work may diminish with time, and the work needs to be published quickly to have an impact.

Q31: Will Findings papers have some form of presentation / poster session in the main conference?

A31: A decision about this will be announced at the conference forum by May 16 and at the conference website shortly after. We are conducting a survey on this, which closes on May 13. An email about the survey was sent to all ACL-IJCNLP authors, the reviewing committee and the ACL mailing list on May 10.

Q32: Were the going-to-Findings decisions biased by factors such as mainstream vs. controversial?

A32: Going-to-finding decisions are made by SACs and PCs, not by reviewers/ACs. The review form does not have any mentioning of Findings.

Q33: Will the Findings affect ACL workshops?

A33: We allow double submissions to the ACL main conference and workshops. If the authors take up the Findings offers, they should withdraw their workshop submissions. In this case, it could potentially reduce the submissions to workshops.

Q34: Will Finding papers be included in the proceedings of the main conference?

A34: No. There will be a separate volume for Findings papers, which will be included in the ACL anthology under “Findings of ACL”.