ACL-IJCNLP 2021

Outstanding Reviewers

2021-07-12T00:00:00-05:00

We would like to advertise outstanding reviewers, listed below

Rami Al-Rfou
Carl Allen,
Mark Anderson,
Stefanos Angelidis,
Jean-Yves Antoine,
Leila Arras,
Rohit Babar,
Hritik Bansal,
Su Lin Blodgett,
Valts Blukis,
Nadjet Bouayad-Agha,
Arthur Bražinskas,
Michael Bugert,
Vittorio Castelli,
Hou Pong Chan,
Fenia Christopoulou,
Elizabeth Clark,
Kevin Clark,
Vincent Claveau,
Anna Currey,
Hal Daume III,
Forrest Davis,
Steve DeNeefe,
Daniel Deutsch,
Sunipa Dev,
Joseph P. Dexter,
Pablo Duboue,
Philip Dufter,
Ondřej Dušek,
Rory Duthie,
Nouha Dziri,
Alexander Fabbri,
Agnieszka Falenska,
Sergey Feldman,
Daniel Fernandez-Gonzalez,
Anjalie Field,
Margret Fleck,
Michael Flor,
Maxwell Forbes,
Thomas Francois,
Daniel Fried,
Zhenxin Fu,
Matteo Gabburo,
Yang Gao,
Siddhant Garg,
Aina Garí Soler,
Marcos Goncalves,
Jana Gőtze,
Bruno Guillaume,
Xiaochuang Han,
Peter Hase,
Hiroaki Hayashi,
Devamanyu Hazarika,
Jack Hessel,
Tsutomu Hirao,
Ari Holtzman,
Xuanjing Huang,
Gabriel Ilharco,
Gilles Jacobs,
Alon Jacovi,
Sarthak Jain,
Nanjiang Jiang,
Anders Johanssen,
Jaap Kamps,
Siddharth Karamcheti,
Brendan Kennedy,
Jihyuk Kim,
Byeongchang Kim,
Nikita Kitaev,
Hayato Kobayashi,
Noriyuki Kojima,
Seth Kulick,
Sawan Kumar,
Adhiguna Kuncoro,
Jake Lever,
Yaoyiran Li,
Jindřich Libovický,
Fangyu Liu,
Wei-Yun Ma,
Adyasha Maharana,
Alexander Mehler,
Sabrina J. Mielke,
Evangelios Milios,
Sewon Min,
Jeff Mitchell,
Matan Orbach,
Jessica Ouyang,
Aishwarya Padmakumar,
Bhargavi Paranjape,
Letitia Parcalabescu,
Carla Parra Escartín,
Viviana Patti,
Karl Pichotta,
Tiago Pimentel,
Lahari Poddar,
Rajkumar Pujar,
Xiaojun Quan,
Shuhuai Ren,
Philip Resnik,
Gil Rocha,
Sylvie Saget,
Victor Sanh,
Timo Schick,
Tyler Schnoebelen,
Roy Schwartz,
Abigail See,
Rico Sennrich,
Peter Shaw,
Qinlan Shen,
Tianze Shi,
Valentina Sintsova,
Wei Song,
Youngseo Song,
Andreas Spitz,
Yoshihiko Suhara,
Alane Suhr,
Ronen Tamari,
Yuanhe Tian,
Rob van der Goot,
Emiel van Miltenberg,
Rik van Noord,
Lucy Vanderwende,
David Vilares,
Alex Wang,
Zijian Wang,
Zhen Wang,
Alex Warstadt,
Gail Weiss,
Alina Wróblewska,
Joern Wuebker,
Jiacheng Xu,
Michael Yoder,
Naoki Yoshinaga,
Steve Young,
Dian Yu,
Wei Zhang,
Zeyu Zhang,
Dong Zhou,
Ran Zmigrod,
Markus Zopf

Call for Diversity & Inclusion Subsidies

2021-06-15T00:00:00-05:00

Diversity & Inclusion Subsidies

ACL-IJCNLP 2021 is providing D&I funds for registration, caregiving, bandwidth and VPN subsidies. The grants are intended for individuals for whom attending the conference would cause a financial burden or create risks to their safety or privacy. Please apply by June 27, 2021 (2 weeks before the early registration deadline) at this link: https://forms.gle/EekTNXekyHeKQpDY9. Informal enquiries can be made to acl2021_div_incl@googlegroups.com.

Registration subsidies: This is for getting a waiver for ACL registration. If you haven’t already registered for an *ACL in 2021, you would also need to pay the ACL membership registration fee, e.g. $100 for regular. You can also request for ACL membership if that’s applicable.

Note that if you are applying for a registration fee waiver, it is expected that you DO NOT register for the conference until you hear from us about your D&I subsidy application. That way, you would not have to ask for a reimbursement, since you would not be charged in the first place. We strongly encourage folks to apply for a volunteership program in order to maximize their chances for getting their registration fees waived.

Caregiving and accessibility subsidies: We can reimburse caregiving purchases that you need for the conferences. You need to provide receipts for your purchases. These can be babysitting, transportation costs, or other support required for yourself or your dependents, which would ease your participation. If you need any personal or technical assistance to access and navigate the conference due to a disability, we can reimburse you for the cost of it. This holds for example if you need assistance in using gather.town due to a visual impairment. We acknowledge that accessibility needs are highly individual to each participant, so please feel free to reach out to us with any issues you might face in this regard.

Bandwidth and VPN subsidies: We can reimburse bandwidth or VPN purchases that you need for the conferences. You need to provide receipts for your purchases. Bandwidth subsidies would be for high-speed internet access costs for the duration of the conference only. VPN generally does not cost more than 10 USD per month and could be applied towards participation if you need anonymity to participate completely (for example, for queer folks who might need this for extra security depending on where they live).

Selection Criteria: Applicants for the subsidy program will be evaluated based on the material they submit in their application packages. Preference will be given to applicants who are presenting a paper in the main conference, the Student Research Workshop, or any of the workshops associated with ACL-IJCNLP, and who do not have other means of support.

Instructions: Applicants for the D&I subsidy Program should fill the application form at: https://forms.gle/EekTNXekyHeKQpDY9

Application deadline: June 27, 2021 (anywhere on earth) – 2 weeks before the early registration deadline

Notification of acceptance: July 4, 2021 – 1 week before the early registration deadline

We aim to send all notifications by the deadline above. However, we will start processing the applications as they come and you may be contacted earlier.

ACL-IJCNLP 2021 Registration

2021-06-10T00:00:00-05:00

Registration Registration is now open! Please register here

We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

Reminders: It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

Registration Types

Regular
Student
I am a Volunteer (student or non-student)
I am a D&I Award Recipient
I am an SRW Award Recipient
I am an invited Conference Organizer (Invited by the General Chair or Local Arrangements)
I am a Staff Member (ACL staff)
I am a Keynote Speaker
I am a Tutorial Presenter
I am an invited Sponsor or Exhibitor (Invited by sponsoring company providing list of their complimentary people)

ACL Membership

The ACL policy is that everyone attending an ACL conference must be a current (CY 2021) ACL member. So, in addition to the above registration fee, each Regular will add $100 membership fee to their registration and each Student will add $50 membership fee to their registration if not already a 2021 paid member. ACL membership is for the entire calendar year, providing membership in the society devoted to promoting CL research throughout the world, open access to the CL and TACL Journals and ACL Anthology as well as access to other ACL conferences in the same year without paying membership again. Membership also entitles you to electronic notification of new issues of the journals, discounts on publications from participating publishers, announcements of ACL and related conferences, workshops, and journal calls of interest to the community.

Reduced Rate Memberships

The ACL offers a discount to members from developing countries. We hope this discount will allow for more people from these countries to become a part of the ACL and enjoy its benefits. To be able to benefit from reduced rate memberships, you need to become a member through the ACL Member Portal. You can then register for the conference as a current member. For more information follow this link: https://www.aclweb.org/portal/content/membership-fees

Virtual Conference Cancellation Policy

For cancellations received up to two weeks prior to the conference, registration fees will be reimbursed; ACL membership (if paid as part of the registration) is not reimbursable.

Call for Diversity & Inclusion Subsidies

Instruction for Camera-Ready Submission

2021-05-14T00:00:00-05:00

May 14, 2020 These instructions are adapted from the ones for ACL 2020, EMNLP 2020, and NAACL 2021.

To submit your camera-ready version, please go to the softconf site, click the submission id, and then click “Final Camera Ready Submission”. This document is organized based on the sections in the Final Submission Page. Please note that the following applies to the ACL-IJCNLP main conference papers and Findings of ACL papers. It will also apply to ACL-IJCNLP workshop papers unless the workshops have their own instructions.

Please remember to use the template files for the conference and follow the instructions in this document. In addition, you need to run a tool called aclpub_check developed by the NAACL2021 publication chairs to automate the checking for some frequently encountered formatting issues. For details of the tool, see Section 3 and the Appendix. Camera-ready copies that fail to meet these instructions will be returned for re-submission.

1. Metadata (Part A in the final submission form)

Can I make changes to the author list in the camera-ready version?

No, you cannot change the author list, as explained in the FAQ at https://2021.aclweb.org/blog/faq-decision/ In fact, the authors’ field will not be editable in the final submission form.

How should I enter metadata on the START system?

The metadata (title, author, abstract) that you enter into START is very important, because it is used on the conference website, proceedings, handbook, mobile app, and the ACL Anthology (and propagates to Google Scholar, etc.).

Before the metadata is entered, please have all authors ensure that the name in their START profile (User Console, Update Profile) appears exactly the way that they want it to appear.

Unicode (UTF-8) can be used for accented or special characters.
Ordinarily, names are NOT written in all caps or all lowercase.
The “Last Name” is the name(s) by which your paper is to be cited. It is usually a family name, even for authors from cultures where the family name is written first.
The “First Name” is usually a given name or names, including middle names/initials.

The metadata should be written using Unicode (UTF-8) with LaTeX commands. Please try to follow these guidelines:

In titles, please capitalize the first word, the first word after a colon (:), and all other words (including hyphenated words like Mixed-Case), except the function words: articles, pronouns, conjunctions, prepositions, particles, subordinating conjunctions, and the infinitive marker “to”.
BibTeX (in many bibliography styles, including ACL’s) lowercases the titles of conference papers, and needs to be told which letters not to lowercase. So if your title has letters that should always be capitals, please protect them with curly braces, like this: {E}nglish, {C}homsky, {IBM}, {CFG}s, {HMM}s. Please also protect the first letter after a sentence-final punctuation mark. For example:

Can {LSTM} Learn to Capture Agreement? {T}he Case of {B}asque Named Entity Extraction from Noisy Input: {S}peech and {OCR}.

It is important to only protect those first letters that belong to the categories exhibited above, i.e., proper names (including language names), acronyms, abbreviations, the first letter after punctuation (such as a colon) and the first letter in a sentence. These curly braces will not appear in the online conference program or proceedings. They will only appear in the BibTeX file that others will use to cite your paper.

If you need literal curly braces, please escape them like this: { }
Please don’t use any nonstandard LaTeX commands, and there should be no \footnote or citations using \cite or related commands.
You can use LaTeX math mode where appropriate: An $O(n^2)$ Algorithm for $n$-gram Smoothing.
You can use Unicode (UTF-8) for accented or special characters.
If you copy-and-paste from your PDF file, please be sure to rejoin words broken by hyphenation.

2. Copyright (Part B in the form)

What about copyright?

When you submit the paper, you will be asked to sign the ACL Copyright Transfer Agreement on behalf of all authors, electronically (via the START Conference Manager). Authors retain many rights under this agreement and it is appropriate in the vast majority of cases. Please contact the publication chairs at acl2021publicationchairs@googlegroups.com, with any concerns regarding copyright.

Who should sign the copyright form?

Before signing this form, please confirm with your co-authors (and, if applicable, your and their employers) that they authorize you to sign on their behalf. Only the authorized representative needs to sign the copyright form. Please sign your full name (not just your first or last initials).

3. Main paper (Part C in the form)

What are the page limits?

For both long and short papers, ACL-IJCNLP 2021 allows one extra page to help address reviewer comments. Please use the extra space to help address reviewer comments. Long papers are permitted at most 9 pages of text and short papers may use up to 5 pages of text, plus unlimited space for references and the impact statement. Acknowledgements count toward the page limit.

What is the max size of the PDF file?

The max size of the PDF file (including the Appendices, see Section 4) should be no more than 20MB.

What is the format for the camera-ready copy?

The file must be in Portable Document Format (PDF) on A4 paper. We require the use of ACL LaTeX style files or Microsoft Word style files tailored for this year’s ACL conference. You can access the style files and detailed formatting instructions here: https://2021.aclweb.org/downloads/acl-ijcnlp2021-templates.zip or https://www.overleaf.com/latex/templates/instructions-for-acl-ijcnlp-2021-proceedings/mhxffkjdwymb. If you are using LaTeX, please create the PDF file with pdflatex or xelatex. This ensures use of the proper Type 1 fonts and also takes advantage of other PDF features. You will have the best results using a modern LaTeX distribution, in particular, TeX live. Using the geometry package to set the A4 format is recommended.

How do I check the format of the camera-ready version before submitting?

You should check the paper format according to this instruction page and use the style template files with the links above. The NAACL2021 publication chairs have developed a package called aclpub_check to automate much of the format checking (See the Appendix for details). Please run this package before submitting your paper to softconf in order not to miss some subtle formatting details.

How should the final copy differ from the original submission?

The camera-ready version of your paper should incorporate the comments of the reviewers as well as other changes you see fit to make. In addition, be sure to do all of the following:

Ensure that your paper conforms to the provided styles, font and page size.
Include the authors’ names and affiliations under the title. Note that the list of authors should be identical to the list specified when submitting the paper and should be in the same order.
De-anonymize references to your own work in the body of the paper.
Where appropriate, add acknowledgments for colleagues, reviewers, and grants. Do not number the Acknowledgements section. Please note that the acknowledgement section should fit within the allowed page limits (9 pages for long and 5 pages for short papers, plus unlimited pages for the impact statement and references) and be in the same font as the rest of the paper.
Ensure that all tables, graphs, and figures are readable at standard resolutions.
The Appendix (if exists) should appear after the references, as part of the same PDF file (see Section 4); in contrast, the supplemental materials should appear as a separate zip file (see Section 5).

What are the tips to make my final version more accessible?

As a central venue of publication for our community, please prioritise the accessibility of your final version. The Diversity & Inclusion committee for ACL 2020 has outlined some tips on how to do this: https://acl2020.org/blog/accessibility-for-camera-ready/

How do I ensure that my file is correctly formatted?

You need to run the aclpub_check package (see the Appendix), and pay special attention to things like the following:

The paper size: Your paper needs to be formatted to A4. Here are a couple of ways to check this:
- Using pdfinfo. The pdfinfo command should include "Page size: 595.276 x 841.89 pts" in its output.
- Using Apple's Preview.app. Open the PDF, and type Ctrl-I. It should report the correct page size.
- Using Adobe Acrobat. Open the PDF, navigate to File, Properties..., Description. The field labeled "Page Size" should read 8.27 x 11.69 inches in.
Embedding fonts: You can check your final PDF with the command "[pdffonts](https://www.xpdfreader.com/download.html) mypaper.pdf" and confirm that all the fonts say "yes" under "emb". START will not let you upload your final PDF otherwise. If you are including graphics with the PDF extension, these files must also have embedded fonts. If your paper uses Asian fonts, they must be embedded in the PDF file so that they can be displayed by non-Asian versions of the PDF reader (Asian versions ship with a larger set of default fonts.)
Long titles: The title should NOT exceed the left & right margins. Fold a long title into multiple lines if necessary.
Margins: The margins of the text area must be reserved as specified in the style file.
Page numbers: DO NOT provide page numbers for your PDF file. The page numbers will be generated automatically while compiling the proceedings.
Consistent Author Names: The author names must be consistent everywhere and be the same as registered to the submission page(s), and have the same spelling style in all accepted papers. This can avoid generating non-unique Author Index entries for authors with more than one accepted paper. Be sure to ask your co-authors whether you enter their names correctly.

What if my paper includes graphics?

Remember that you are providing a camera-ready copy. Thus, artwork and photos should be included directly in the paper in their final positions. Ideally, you should use vector graphic formats (PDF, EPS), which allow the graphics to scale arbitrarily. Avoid GIF or JPEG images that are low resolution or highly compressed.

Your paper must look good both when printed (A4 size) and when viewed on screen as PDF (zoomable to any size, color okay). Thus, you may want to use color high-resolution graphics, allowing readers to zoom in on a graph and study it. However, please check that the same graph or photograph is legible when printed and in a PDF viewer at different resolutions. Don’t go overboard on resolution; keep file sizes manageable. Note that vector graphics (e.g., encapsulated PostScript) look good at any scale and take up little space (unless you are plotting many thousands of data points).

What if my paper’s title or abstract has changed?

In addition to edit the title/abstract in the main paper, please also remember to edit those metadata fields when you upload the camera-ready version, so that they will appear correctly in the table of contents, author index, conference schedule, etc. Please also note that your name will appear in conference metadata as you have configured it in START, so make sure that it is correct there (e.g., capitalization, full name, etc.). You can change this on the user settings page of the START conference manager, under “User” -> “Account Information” -> “Update Profile” .

4. The Appendices (As the last part of the pdf file for Part C in the form)

Appendices are for things such as lemmas, hyperparameters, formulas, proofs, and tables that are informative but not critical to the understanding of the paper.

Where do the Appendices go?

Appendices should be part of the PDF file uploaded to Part C in the form. It should appear at the end of the PDF file, after the references.

What’s the page limit of the Appendices?

The Appendices do not count towards the page limit of the main paper, but they need to be no more than 4 pages, as recommended by the publication chairs.

What template should the Appendices use?

The Appendices should use exactly the same template as the main paper (See Section 3).

5. Additional document for conditionally accepted papers (Part D in the form)

Who needs to submit a document for Part D?

That depends on the status of your paper:

If the status is Accept-P1-condition or Accept-P2-condition, you must submit an additional document (explained below).
If the status is Accept-P1 or Accept-P2, please do NOT upload any documents to Part D even if your paper receives an EAC review and/or an EAC metareview. You can skip this section.

What should the document look like?

In this document, authors need to provide a short document that explains how they have made the changes requested by the EAC meta-reviews (which appears at the bottom of all other reviews and has a special heading). The document can be either a plain text file or a PDF file, and should list each of the concerns from the original EAC meta-review along with a short summary of what was done to remedy the concern.

To make this as straightforward as possible, we recommend copying and pasting the contents of the EAC meta-review into the document and then describing the action taken after each item in the EAC metareview. If there are additional issues mentioned in EAC reviews that were not included in the EAC metareview, authors may choose to address those points as well; however, only the points mentioned in the EAC meta-review will be the basis of whether the final version of the paper will be accepted. This explanatory document will not be made public; it is just to expedite the conditional acceptance workflow for the EAC’s review process.

The size of this document should be no more than 10MB.

What will happen after June 1?

The EAC chairs will go over this document and check the camera-ready version to determine whether the required changes have been included in the camera-ready version. If so, the condition will be removed, the paper will be accepted to the main conference or Findings, and the status of having been conditionally accepted will not be publicly visible. If not, the paper will be rejected. The PCs will inform the authors of the final EAC decisions by June 7, 2021.

If you have any questions about EAC reviews or the changes required by EAC, you can contact the ACL-IJCNLP 2021 Ethics Chairs acl-2021-ethics-chairs@googlegroups.com. You can also check the Ethics FAQ at https://2021.aclweb.org/ethics/Ethics-FAQ/.

6. Supplementary material (Part E in the form)

Where do supplemental materials go?

Supplemental material contains material such as data, software, all other material and is uploaded separately as a zip file. The maximum size of the zip file is 20MB.

Do I need to submit latex source files?

No. Latex files will not be used in ACL Anthology, thus please do NOT submit them.

7. Deadlines

When and where do I send my final camera-ready paper?

You must submit the final version of your paper by June 1, 2021 (11:59pm, UTC-12 hours, “anywhere on Earth”) by navigating to the ACL 2021 START login page and following the internal links. This is a firm deadline, i.e., no change will be accepted after that.

8. Questions or Comments

If you have any questions about:

the camera-ready version including running the aclpub_check package: please contact our publication chairs at acl2021publicationchairs@googlegroups.com.
the EAC issues and additional document in Part D: please contact the EAC chairs at acl-2021-ethics-chairs@googlegroups.com

Please do NOT contact ACL-IJCNLP PCs unless your questions cannot be answered by the publication chairs or the EAC committee.

Appendix: Checking the format with the aclpub_check package

This appendix is modified from the document of the NAACL 2021 publication chairs at: https://docs.google.com/document/d/1MYKGS8x17Mvl4fKYz_ZHIkre9KBhzB_9W87bu4rbtqI/edit

Following NAACL 2021, we are using a package called “aclpub_check” that automatically detects author formatting errors, margin violations as well as many other common formatting errors. Before submitting your paper to softconf, please run the package first and fix the detected errors.

A.1: Access the package

The package is written in Python and there are four steps to using it:
1: git clone https://github.com/yz-joey/ACLPUB.git
2: cd ACLPUB
3: pip install -e .
4: python3 aclpub_check/formatchecker.py –paper_type PAPER_TYPE PAPER_NAME.pdf

You should also be able to use it via:
1: pip3 install git+https://github.com/yz-joey/ACLPUB.git
2: python3 -m aclpub_check.formatchecker –paper_type PAPER_TYPE PAPER_NAME.pdf

PAPER_TYPE should be the word short or long. PAPER_NAME can be any valid file name.

Below, we are walking you through how to use the tool by considering three papers published by publication chairs at NAACL 2021.

A.2. Example 1

First, consider Josef Valvoda’s paper “What About the Precedent: An Information-Theoretic Analysis of Common Law”

python3 aclpub_check/formatchecker.py –paper_type long precedent.pdf

Checking precedent.pdf
Error (Margin): An image on page 1 bleeds into the margin.

We detected 1 error and 0 warnings in your paper.

Thus, we see this paper has a margin violation. The script output a png called “errors-precedent-page-1.png” which you can see below:

It shows that Josef has to shrink the first-page picture to make the paper compliant with the NAACL formatting rules. This is easily remedied with the adjustbox https://www.ctan.org/pkg/adjustbox package or a similar bit of LaTeX.

A.3: Example 2

Next, consider Tiago Pimentel and Irene Nikkarinen’s paper: “How (Non-)Optimal is the Lexicon?”

python3 aclpub_check/formatchecker.py –paper_type long lexicon.pdf

Checking lexicon.pdf
Error (Margin): Text on page 13 bleeds into the margin.

This time, the automatically generated png shows me that the authors didn’t keep some math equations out of the margin in a proof in the appendix.

Here, it’s likely that the equations have to be broken over two lines.

A.4: Example 3

Finally, consider Jennifer White’s paper “A Non-Linear Structural Probe”

python3 aclpub_check/formatchecker.py –paper_type short structural.pdf

Checking structural.pdf

All Clear!

So, there were no mistakes!

The script checks for many violations, e.g. page-limit violations, font and font size violations and even a few common typos. The script even makes recommendations about citing non-arXiv versions of papers. These, however, are simply warnings:

Warning (Bibliography): It appears you are using arXiv links more than you should (18/55). Consider using ACL Anthology DOIs instead.

FAQ for the Decision Process, Camera-ready Version, and Findings

2021-05-10T00:00:00-05:00

May 10, 2021

The whole review process is quite complicated, and may be mysterious to many authors. Thus, we want to explain the process a little bit in Section 1 before answering more specific questions.

Section 1: The Review Process

Below are the main steps in the review process, which is very similar to those from previous conferences:

2/2-2/24: Paper check and AC/reviewer assignment
1. Papers are assigned to tracks by Program Chairs (PCs) and checked by Senior Area Chairs (SACs) and PCs. Let's call these tracks the technical tracks, in contrast to the EAC track (see Step 3).
2. Three reviewers and one area chair (AC) are assigned to each paper.
2/25-3/20: Reviewers review papers
1. Reviews include two parts: non-confident part (which was sent to authors on March 25), and confident part visible only to ACs/SACs/PCs.
2. If the reviewers or the AC had ethical concerns about the paper, they would "flag" the paper by clicking "yes" for the ethical question on the review form, and provide a justification for their concern.
3/21-3/24: EAC check and reviewer assignment
1. PCs add the flagged papers to a special track for the Ethic Advisory Committee (EAC).
2. EAC chairs then read the paper to determine whether it needs a full EAC review. If so, one or more EAC reviewers are assigned to the paper, and the EAC reviews would be due in mid-April.
4/1-4/7: Reviewer/AC discussion period
1. The AC and reviewers of technical tracks discuss the paper in softconf's discussion board. The reviewers are asked to update their reviews in softconf if they change their opinions or have any new comments to add about the paper based on the discussion.
4/8-4/14: The AC writes a meta review which has three parts:
1. AC recommendation, which uses one of four labels: 1-Accept, 2-Possible-Accept, 3-Possible-Reject, 4-Reject.
2. AC summary of reviews (including his/her own opinion) of the paper, which is visible to authors.
3. AC's confidential comments to SACs/PCs.
4/15-4/24: SAC recommendation which also has three parts:
1. SAC recommendation, which uses one of four labels: 1-Definite-Accept, 2-Possible-Accept-to-Main-Conference, 3-Possible-Accept-to-Findings, 4-Reject.
2. SACs provide a ranking of papers with the 2-Possible-Accept-to-Main-Conference or 3-Possible-Accept-to-Findings recommendations.
3. SACs' comments to PCs
3/25-4/24: EAC recommendation:
1. EAC reviews are due in Mid-April.
2. EAC chairs read the EAC reviews and inform PCs of their decisions, which include comments and one of three labels:
  1. E1: The paper does not have serious ethical issues
  2. E2: The paper has some serious ethical issues and thus, if the paper is accepted, the camera-ready version needs to be checked by EAC to ensure ethical concern has been addressed in that version.
  3. E3: The paper should be rejected due to ethical concerns.
4/25-5/5: PCs make final decisions:
1. Based on the input from the technical tracks (reviews, discussion board, AC recommendation, SAC recommendation, etc.), PCs use one of three labels for each paper:
  1. P1: Accept-to-Main-Conference
  2. P2: Accept-to-Findings
  3. P3: Reject
2. For P1 or P2 papers (not P3 papers), EAC chairs write a meta review that highlights the changes that EAC recommends authors to make in the camera-ready version.
3. For P1 or P2 papers, PCs will use the following four labels for the final decisions:
  1. If EAC recommendation is E1, the paper is accepted and the final label is Accept-P1 or Accept-P2.
  2. If EAC recommendation is E2, the paper is conditionally accepted, and the status is Accept-P1-condition or Accept-P2-condition.
  3. None of the papers get an E3 label from the EAC.
4. For P3 papers, PCs will mark them as Reject-P3.
5. PCs send out notifications to all authors.

Now, if you go to softconf, you can see your papers in one of five categories:

Accept-P1: accepted to the main conference
Accept-P1-condition: conditionally accepted to the main conference
Accept-P2: accepted to Findings
Accept-P2-condition: conditionally accepted to Findings
Reject-P3: reject

In summary, the final decision is based on the input from all the steps mentioned above:

The submission itself and the author responses
Reviews from 3+ reviewers: including the part visible to authors and confidential comments to ACs/SACs/PCs.
AC recommendation: including a recommendation label, AC summary, and confidential comments to SACs/PCs.
Reviewers/AC discussion messages in the softconf discussion board.
SAC recommendation: including a recommendation label, rank of the papers in the area, and comments to PCs.
EAC recommendation: including an EAC label (E1, E2, E3), EAC comments confidential to PCs, one or two EAC reviews, and an EAC meta review for accepted or conditionally accepted papers.

In addition, we target an acceptance rate of the main conference and the Findings to be around 21-24% and 15%, respectively.

Section 2: Questions about the review process

This section includes questions that may be relevant to more than one category.

Q1: How does the review process work?

A1: See Section 1

Q2: The AC recommends Accept. Why is my paper still rejected?

A2: As explained in Section 1, AC recommendation is just one of many factors used in the final decision. For instance, some ACs disagree with reviewers: they may recommend Accept when some reviewers insist on the paper being rejected, or the other way around. PCs also look at other factors such as author rebuttal, three reviews, the discussion in the discussion board, SAC recommendations, EAC recommendations (for flagged papers), and the submission itself.

Q3: My overall recommendation scores are 4, 3.5, 3.5. Why is my paper still rejected (or accepted to Findings, but not to the main conference)?

A3: Just like in Q2, review scores are just one of many factors considered by PCs. For instance, some ACs point out that in some reviews, the overall recommendation scores do not match the reviews (e.g., a largely negative review with a score of 3.5), but reviewers do not want to change their reviews or overall recommendation scores for various reasons, even when prompted by ACs. In that case, PCs read reviews, AC recommendation and SAC recommendation in order to reach a final decision.

Q4: My reviews are largely positive but the paper is rejected due to an EAC review. Does EAC have the one-vote veto right?

A4: No. EAC recommendation is only one piece of information used in the final decision, as explained in Section 1.

Q5: Where can we see revised reviews (or comments to our rebuttal)?

A5: The reviews sent to authors on May 5 are the revised reviews. If reviewers had some comments to the rebuttal that they wanted the authors to see, they would include that in their revised reviews.

Q6: How come the reviews remain the same after we spent so much time writing the rebuttal?

A6: Many reviewers were discussing the papers in the discussion board at softconf, and, after the reviewer/AC discussion, they decided to keep their original reviews because they were not convinced by the authors’ rebuttal. In that case, the new reviews will be identical to the original ones.

Q7: Some reviewers (or the AC) misunderstood our paper. Please reconsider.

A7: If some (but not all) reviewers or the AC misunderstood your paper, the overall recommendation scores tended to vary a lot among reviewers or AC’s meta review would disagree with the three reviews. SACs and PCs paid special attention to such papers, and the final decision is the result of considering all the input including AC and SAC recommendations and the discussion board.

Q8: Can we find out whether the reviewers of my paper had read my rebuttal?

A8: The review form contains a question asking reviewers whether they have read the rebuttal. We are collecting the statistics on that question and can release that statistics later if needed. However, we do not plan to release that information for individual papers for two reasons:

Some reviewers read the rebuttal, discussed the paper in the discussion board, but for some reasons did not answer that question in the review form. Thus, the answer for an individual paper might not be very accurate.
When PCs created the review form in Jan 2021, this question was marked as visible to ACs/SACs/PCs only. Thus, PCs should not change the visibility setting of the question after the fact.

Q9: I have questions about EAC review or meta review. How can I get some clarification?

A9: Please read the Ethics FAQ first at https://2021.aclweb.org/ethics/Ethics-FAQ/. If you still have questions, contact the EAC chairs at acl-2021-ethics-chairs@googlegroups.com directly.

Q10: After the rebuttal, some overall recommendation scores decreased. Why?

A10: This is likely to be due to the reviewer/AC discussion. Some reviewers’ opinions could change due to the discussion or after reading authors’ rebuttal, so the scores can go up or down in the revised reviews. The change of the scores shows that the reviewers have read the author rebuttal.

Reviewers were asked to revise their reviews to reflect their new opinion of the paper, but that might not always happen. Unfortunately, we cannot make the information in the discussion board available to the authors due to the nature of double-blind reviews.

Q11: The AC raised some issues that were not found in the three reviews, which we could not address in the rebuttal period. We disagree with those points.

A11: ACs can point out issues that were not identified by the reviewers. That is why AC recommendation was one of the factors for the final decision. As for your disagreement with AC’s opinion, please see A7 above.

Q12: My paper is marked as accepted to the main conference (or the Findings), but it includes an EAC review and a EAC meta review. Does that mean my paper is conditionally accepted instead?

A12: If a flagged paper was determined by the EAC that a full EAC review was necessary, the paper would get one or EAC reviews. If such a paper was accepted to the main conference or Findings, an EAC meta review would be added. Thus, having those reviews does not mean that the paper is conditionally accepted. If you want PCs to double check for you, please email PCs with your submission ID.

Q13: Some reviewers raised new issues in the revised reviews which we have no chance to respond to. Had they mentioned that in their original reviews, we could have addressed their concern.

A13: It is not uncommon for new issues to be raised during the reviewers/AC discussion period, and reviewers can include such issues in their revised reviews. Given the short review cycle, it is impossible to have another round of author rebuttal after the discussion period. Nevertheless, as mentioned in Section 1, reviews are just one of many factors considered in the final decision.

Q14: Now that my paper has been accepted to the Findings, can I post it to arXiv.org now?

A14: Yes, that should be fine.

Q15: For some reasons I have not received an accept/reject notification from you. What’s the status of my paper?

A15: Please go to the softconf site, click on your paper, you should be able to see the status and reviews of your paper.

Section 3. Questions about the camera-ready version

Q16: Can I make minor changes to the title in the camera-ready version?

A16: Yes, minor changes to the title are allowed.

Q17: Can I make any changes to the author list in the camera-ready version?

A17: No, you cannot make any changes to the author list. The authors’ information was used for COI (conflict-of-interest) detection, and COI is a factor that affects many stages in the review process (e.g., AC/reviewer assignment). Thus, no changes to the author list is allowed in the camera-ready version. In fact, in the final submission form, the author field is not editable.

Q18: Where can we find the templates for the camera-ready version?

A18: We are working with publication chairs on this, and will contact authors above this once the templates are available.

Q19: I don’t see the final submission page. Why?

A19: The Final Submission Page will be open by May 14, 2021 for authors to start uploading their camera-ready version. Before that deadline, the authors will NOT be able to see the camera-ready submission link at softconf.

Q20: What is the page limit for the camera-ready version of the paper?

A20: You will have one extra page for that, 9 pages for long, and 5 pages for short papers, plus unlimited space for references and the impact statement.

Q21: Should the appendix be part of the main paper? What’s the page limit for the appendix? What style file should I use for the appendix?

A21: We are checking with our publication chairs on this. Stay tuned.

(We leave some space for Q22-Q25 here, which will be added once we hear back from publication chairs)

Section 4. Questions for Findings

This section is modified from the Finding FAQ from EMNLP 2020 at https://2020.emnlp.org/blog/2020-04-19-findings-of-emnlp

Q26: What criteria are used to determine whether a paper should be accepted to the main conference or Findings?

A26: The low acceptance rate for the ACL-IJCNLP 2021 main conference means many high-quality work would have to be rejected without the Findings category. That’s the main motivation for us to have this category. Thus, Finding papers are ones with solid work but cannot make it to the main conference due to the low acceptance rate.

Q27: Does Finding have its own ISBN or DOI?

A27: We expect that Findings will grow with future conferences (e.g., having a new volume associated with each conference), and become a recognised and respected publication venue within the field. Initially, we will not be seeking to have Findings indexed in Scopus and Web-of-Science etc. - this is a complex process, and TACL is only now applying after 7 years of operation - but this is a natural option we might consider in future years.

Q28: Why might you want your paper to appear in Findings?

A28: Those with papers accepted to Findings will have the option of having the work accepted largely as-is (with the exception of Accept-P2-condition papers), and being made available online promptly. This will be an attractive option for work that is particularly time-critical, e.g., the novelty of the work would be diminished if publication were delayed by being rejected from ACL. This will also help for papers caught by overlapping anonymity periods between conferences, which can preclude public release for a long period. In this case, Findings can offer the option of a quick publication. Note that authors whose papers are accepted for publication in Findings can, of course, opt out, by withdrawing or simply not submitting a camera-ready copy of the paper.

Many papers languish on arXiv without ever being published in a peer-reviewed venue because they were overtaken by the accelerating pace of NLP research. This new publication option will give more authors the chance to publish papers that are good and decent contributions at a given point in time, even if their long-term impact may be modest. This will also reduce the amount of unreviewed work in our field, and thereby increase quality.

Q29: Is there any advantage of publishing in Findings versus an ACL workshop?

A29: This depends on the publication prestige, which we would expect to be higher for Findings than for most workshops. Moreover, workshops increasingly tend to move to non-archival format.

Q30: Why should the authors agree to publish in Findings if they know the paper was only narrowly rejected from ACL, and so might get into another conference?

A30: This is a judgement call to be made by the authors. Some authors may want to take the acceptance, rather than risk another rejection. This might be more pertinent if the paper has already been rejected from another publication venue, or if the author needs to fill out their CV, e.g., a PhD student wanting to secure a postdoc or faculty position. Other reasons might be that the importance of the work may diminish with time, and the work needs to be published quickly to have an impact.

Q31: Will Findings papers have some form of presentation / poster session in the main conference?

A31: A decision about this will be announced at the conference forum by May 16 and at the conference website shortly after. We are conducting a survey on this, which closes on May 13. An email about the survey was sent to all ACL-IJCNLP authors, the reviewing committee and the ACL mailing list on May 10.

Q32: Were the going-to-Findings decisions biased by factors such as mainstream vs. controversial?

A32: Going-to-finding decisions are made by SACs and PCs, not by reviewers/ACs. The review form does not have any mentioning of Findings.

Q33: Will the Findings affect ACL workshops?

A33: We allow double submissions to the ACL main conference and workshops. If the authors take up the Findings offers, they should withdraw their workshop submissions. In this case, it could potentially reduce the submissions to workshops.

Q34: Will Finding papers be included in the proceedings of the main conference?

A34: No. There will be a separate volume for Findings papers, which will be included in the ACL anthology under “Findings of ACL”.

Accptance Decision

2021-05-05T00:00:00-05:00

Notes about the Accept/Reject Notifications

1. ACL Findings:

To continue the success of Findings at EMNLP 2020, the ACL-IJCNLP 2021 reviewing committee has selected papers that are not accepted for publication in the main conference, but nonetheless have been assessed by the Program Committee as solid work with sufficient substance, quality and novelty to Findings. Therefore, the ACL decisions will have the following three basic types : Accept-to-Main-Conference, Accept-to-Findings, and Reject. Some accepted papers are conditional. Please refer to Sections 2 and 3 below for more information.

The conference received 3350 full-paper submissions, and the acceptance rates of the first two types are 21.3% and 14.9%, respectively.

2. EAC Reviews:

The consideration of the ethical impact of our research, use of data, and potential applications of our work has always been an important consideration and, as artificial intelligence is becoming more mainstream, these issues are increasingly pertinent. This year, a new Ethics Advisory Committee (EAC) was formed to look into the submissions flagged for ethical concerns by the reviewing committee (reviewers, area chairs, and senior area chairs of technical tracks).

For any paper that the technical reviewing committee expressed ethical concerns, the paper was sent to the EAC. The EAC chairs went over the papers to determine whether a full EAC review would be required. If so, the paper received one or two ethics reviews in addition to the reviews from the technical tracks. The ethics reviews can be distinguished from technical reviews as they do NOT have a score for Overall Recommendation.

For any paper that is accepted into the main conference or Findings based on technical reviews and that had been referred to the EAC, the EAC chairs have recommended one of the following to the Program Chairs (PCs): (a) accept, (b) conditional accept (the ethical issues must be addressed in the camera-ready version), and (c) reject due to ethical issues.

3. Five Categories for Accept/Reject Decisions:

The PCs combine the EAC recommendations with the recommendations from technical tracks, and use the following five categories for the accept/reject decisions:

Accept-to-Main-Conference
Conditional-Accept-to-Main-Conference
Accept-to-Findings
Conditional-Accept-to-Findings
Reject

Notice that the decisions are not based on the review scores alone. PCs also consider author rebuttal, area chair (AC) recommendations, senior area chair (SAC) recommendations, reviewer discussion, EAC recommendations, and so on. Thus, it is possible that some accepted papers’ review scores are lower than those of some rejected papers.

4. What will happen to the conditionally accepted papers?

All conditionally accepted papers have an ethics meta-review which appears at the bottom of all other reviews and has a special heading. This meta-review states the changes needed to be made by the authors in the camera-ready version. The authors need to submit the camera-ready version by June 1, 2021, the same deadline for the papers without conditions. In addition, the authors need to provide a short document that explains how they have made the changes requested by the EAC meta-reviews. The document can be either a plain text file or a pdf file, and needs to be uploaded to Part D in the Final Submission Page. To be clear, this explanatory document will not be made public; it is just to expedite the conditional acceptance workflow for the EAC’s review process.

The PCs will inform the authors of the decisions by June 7, 2021.

5. The Deadlines:

If your paper belongs to the first four categories, here are the upcoming deadlines for authors and PCs:

May 14, 2021: The Final Submission Page will be open for authors to start uploading their camera-ready version. Before that deadline, the authors will NOT be able to see a link at softconf for uploading the final version.
May 16, 2021: PCs will determine and inform authors how the Finding papers will be presented, after conducting a survey and discussing that with ACL execs, the conference general chair and workshop chairs. The details of the survey will be sent to the ACL mailing list and all the authors and the reviewing committee of this conference in the next few days.
May 25, 2021: The deadline for turning down the “offer”:
- For all the four Accept* categories, should you wish to turn down the offer, you will need to withdraw your paper in the START system by May 25, 2021. If you do not, we will assume that you accept the “offer”.
- If you want to keep your paper in the main conference or Findings, but you have double submitted it to ACL-IJCNLP 2021 workshops, you need to withdraw your paper from the workshops by the same deadline.
June 1, 2021: The camera-ready version is due.
- In addition to the camera-ready paper, the conditionally accepted papers need to submit a separate document that explains how the authors have made the changes required in the EAC meta reviews (see Section 4).
June 7, 2021: PCs will inform the authors of conditionally accepted papers whether the condition on their papers has been removed, based on EAC recommendations.
June 10, 2021: If a paper is accepted to the main conference, the authors will receive an email informing them of the presentation mode of their papers (oral vs. poster) and the details for each mode.
June 24, 2021: The deadline for uploading your presentation videos for accepted papers. More detail about this will be sent out soon by PCs or our virtual conference committee chairs.
August 2-4, 2021: The Main Conference takes place!

Reviewing Advice

2021-04-01T00:00:00-05:00

Reviewing Advice for ACL-IJCNLP 2021

Adapted from the reviewing advice from NAACL-HLT 2021 and the reviewing advice from EMNLP 2020.

The intention for this page is to provide some advice to reviewers, such that we can identify the best research to be presented in the conference, and provide constructive feedback in order for authors to further improve their papers. We recognize and appreciate the amount of efforts reviewers have contributed, and hope to make that more beneficial. We want all the authors to feel the delight when they read the peer reviews for their papers.

To help our reviewers with common review questions, our Reviewer Mentoring Committee has created several videos and posted here. In addition, many major conferences have included advice to reviewers, in NLP and in other fields, and there’s also plentiful advice relating to journal reviewing. Within the field of NLP, we would highlight:

Discursive advice in ACL 2017 from leading lights in the field: Mirella Lapata, Marco Baroni, Yoav Artzi, Emily Bender, Joel Tetreault, Ani Nenkova, and Tim Baldwin
Two example good reviews from NAACL 2018 presented in their reviewing form
A podcast by Noah Smith about peer reviews

Please take the time to look through these excellent resources.

We hope reiterating some dos and don’ts here can help reviewers as well as authors.

First, evaluate the paper’s contributions. This is where you will use your NLP domain knowledge. We advise that you should not accept papers just because their reported results are better, or that they appear to be mathematically sophisticated. These are not sufficient or necessary to constitute contributions. And we advise that you should not reject papers just because their results are not better than state-of-the-art. In the previous *ACL conferences, some reviewers placed too much emphasis on SOTA performance, giving low scores to any systems that failed to reach that. While we aim to publish the very best work, a more constructive question to ask is “state of which art?”. As discussed in this blog post, a paper could offer a step forward in terms of efficiency, generalizability, interpretability, and many other criteria. A convincing contribution of any kind should not be rejected only for not topping the leaderboards.

Regarding different kinds of contributions, here’s what Prof. Philip Resnik at University of Maryland says:

I think there would be significant value in encouraging reviewers to think explicitly about the nature of the contribution, and what questions then need to be asked. As a first pass for consideration/discussion:

Is this research making a scientific contribution? If so:

What is the phenomenon in the world that the authors are seeking to improve our understanding of?

What do we now know about this phenomenon that we did not know before?

Is this research making an engineering contribution? If so:

What is the real-world problem (or set of problems) that this work is making progress on solving?

Alternatively, if it’s not targeting a current real-world problem, what real-world problem(s) will this work help enable solutions of?

Is this research making a theoretical (e.g. mathematical) contribution? If so:

What do we know now that we did not know before?

How does this theoretical or mathematical advance connect to either scientific or engineering goals? (See above.)

Work in computational linguistics might include a mixture of scientific, engineering, and theoretical contributions, rather than just one. But, I am suggesting, if a paper does not make a contribution in any of those three categories, with the sub-bullets having understandable answers, one should seriously consider whether it belongs at the conference.

Second, consider these other important points when reading the paper and writing your review:

Check what the paper’s claims are, and how the content of the paper supports that claim. If the paper claims X and there is a performance increase, is that really because of X?
Be specific in your comments. For example, if you think the authors have neglected to cite key papers, then provide these references in your review. It might be obvious to you, but it’s often less clear to the authors. Being specific will help the authors to formulate a cogent response to the review, and to fix these problems in their paper. And it is worth noting that the authors are not obliged to cite or draw comparisons with contemporaneous work (i.e., appearing within 3 months of submission), especially if it is not published in a peer-reviewed venue.
Be constructive in your advice. Stating that some aspect of the paper is done badly can be helpful in the gatekeeping aspect of review (providing grounds for rejection), but it tends to be less helpful to the authors. Some suggestions of how the authors might improve these problematic aspects can allow them to develop the work into something considerably better.
Be kind in your language, even when being critical. It’s easy to get carried away, and write something nasty that you would never say to someone’s face. Try to be polite in your feedback.

Finally, it’s becoming more common for people to share reviews on social media, especially when the reviews reject the work on spurious grounds. These lead us to advise that the following are often invalid bases for rejecting a paper:

The paper’s language or writing style. Please focus on the paper’s substance. We understand that there may be times when the language or writing style is so poor that reviewers can not understand the paper’s content and substance. In that case, it is fine to reject the paper, however you should only do so after making a concerted effort to understand the paper.
The paper’s work is on a language other than English. We care about NLP for any language.
The paper’s results are not better than SOTA. Please look at the paper’s contributions and findings, as discussed above and in this blog post.
The paper does not use a particular method (e.g., deep learning). No one particular method is a requirement for good work. Please justify why that method is needed. Think about what the paper’s contributions are, and bear in mind that having a diversity of methods used is not a bad thing.
The paper’s method is too simple. Our goal is not to design the most complex method. Again, think what the paper’s contributions and findings are. Often the papers with the simplest methods are the most cited. If a simple method outperforms more complex methods from prior work, then this is often an important finding.
The paper’s topic is narrow or outdated. Please be open minded. We do not want the whole community to chase a trendy topic. Look at the paper’s contributions and consider what impact it may have on our community.
The paper’s topic is completely new, such that there’s no prior art or all the prior art has been done in another field. We are interested in papers that tread new ground.
The paper is a resource paper. In a field that relies on supervised machine learning as much as NLP, development of datasets is as important as modeling work. This blog post discusses what can and cannot be grounds for dismissing a resource paper.

Please refrain from using the reasons above as primary grounds for rejection when writing your reviews. We will ensure authors are aware of these guidelines and can reference them during the author rebuttal period. ACs will be checking reviews carefully based on the above criteria, and may ask that you revise your review, or that you provide objective reasons to justify your positions.

We hope these tips are helpful to reviewers, and hope there will be more authors that appreciate the insightful feedback they get from the reviews, and fewer frustrating authors that complain about review quality.

Additional Resources

To read more about the general advice on reviewing, we recommend the following resources:

NeurIPS not only instructs reviewers as to what to include in their reviews, but also gives examples of useful reviewer comments in their Reviewer Guidelines, organized by evaluation criterion, such as “Contributions of the submission”, “Quality of the submission”, “Clarity”, “Originality”, and “Significance”. This will be particularly useful for new reviewers, and also for authors.
ICML gives some examples of good reviews. Please refer to the Part 2 of their Reviewer Guidelines.
A blog article by Pat Thomson about journal review, which shares a broadly similar procedure to conference review. The article is split into three parts, covering how to read the paper critically, how to decide on revisions required and recommendations to the PC, and how to write constructive feedback.
Wiley Publishing has a cute video with 10 tips for first-time reviewers. A list of those tips can be found in the pdf document. One tip that is useful for EVERYONE is tip 8: “Look at the Conclusion First”. Wiley advocates doing so because the Conclusion will give you a good idea whether the research is an exciting development within its own field. But another reason for doing so is to see what the paper is claiming to have done: This often differs from the Abstract and Introduction, which may make more impressive claims than the work actually supports.
Elsevier’s “Researcher Academy” has produced a video about “How to write a helpful peer review report”, presented by Zoe Mullan, Founding editor and Editor-in-chief of the open access journal “The Lancet Global Health”. To watch the video, you have to sign in to join the academy (which is free).
Some interesting findings about paper reviews can be found in an early article published in the Journal of the American Medical Association (JAMA), “What Makes a Good Reviewer and a Good Review for a General Medical Journal?”.

What any of the articles and videos indicated above will give you is CONFIDENCE that you’re doing the right thing. (Early Career reviewers will be happy to hear that one of the conclusions of the JAMA paper is that “Younger age also was an independent predictor for editors’ quality assessments”)

Instructions for Reviewers

2021-02-22T00:00:00-06:00

ACL-IJCNLP 2021 Instructions for Reviewers

Thank you for agreeing to help in the reviewing process for ACL-IJCNLP 2021! In order to ensure that all valid submissions receive a fair and thorough review, we would like to bring your attention to a few guidelines for reviewing papers. Please read these instructions before you start reviewing papers. All reviewers are assigned no more than 6 reviews each. Since we have a joint long and short paper deadline, please be mindful that you may have a combination of long and short papers to review.

Important Dates

Your reviews are due on Saturday, March 20, 2021 (11:59pm anywhere on Earth). The rebuttal period starts on March 25, thus it is crucial that you complete your assignments by March 20.

Please note that there is a reviewer discussion period from April 1 to 7 after the author response. Your duties are marked in bold. Don’t leave reviewing to the last minute!

Review Period	February 25, 2021 - March 20, 2021
Author Response Period	March 25, 2021 - March 31, 2021
Reviewer Discussion Period (ACs lead discussion)	April 1, 2021 - April 7, 2021

Confidentiality

Please note that the content of any submission to ACL-IJCNLP 2021 and the content of discussion about submissions, are confidential.

The Review Form

We adapt the review forms and the instructions from ACL 2020, EMNLP 2020 and NAACL 2021. The review form consists of four main sections.

1. In-Depth Review. This section is for you to give your overall assessment of the paper and to provide evidence to support your opinions. There are ten subsections:

The core review: This is the most important part. It should include your view of the main contributions that the paper intends to make and how well it succeeds at making these contributions. From your point of view, what are the significant strong and weak parts of the paper and the work it describes? This could be a 2 paragraph (or longer) essay and/or a bullet list. Remember to describe how the work advances the state of knowledge in computational linguistics and/or highlights why it fails to make a sufficient contribution.
Reasons to accept: Please briefly summarize from your core review the main reasons why this paper should be accepted for the conference, and how the ACL community would benefit from it. You may refer back to your review to provide more context and details.
Reasons to reject: Please briefly summarize the main reasons that this paper cannot be published and presented in its current form. What are the parts that would need to be improved in order to advance the state of knowledge?
Reproducibility: Following EMNLP 2020 and NAACL 2021, we provide a reproducibility checklist in an effort to increase reproducibility of the research work in NLP. In the review form. Please answer the following two questions.
- “How do you rate the paper’s reproducibility? Will members of the ACL community be able to reproduce or verify the results in this paper?” Scores of 1-5 are used to assess this aspect. The detailed explanation for each point level is provided in the review form. N/A may be used for the papers that do not include empirical results.
- “Are the authors’ answers to the reproducibility checklist useful for evaluating the submission?”. Three choices are provided for this question (Very useful, Somewhat useful, and Not useful). Note that this question is for us to collect feedback regarding the usefulness of the reproducibility checklist, and is not about evaluating the paper itself.
Ethic review: This year, authors are allowed extra space after the 8th page (or the 4th page for short papers) for a broader impact statement or other discussion of ethics. Regardless of whether the paper includes an impact statement, you need to read the whole paper carefully and determine whether the paper has some potential ethical issues. If that is the case, you should flag the paper by choosing “yes” to the “ethical concerns” question. The flagged papers will then be reviewed by an Ethics Advisory Committee (EAC).
- Please read the FAQ for ethical discussion standards and the questions in the ethics section of the review form for more guidance on some problems to look out for and key concerns to consider relative to the code of ethics.
- If you flag a paper, it is very important that you provide justification as detailed as possible to help EAC to understand your concern.
- Note that more than 30% of the submissions include ethics/impact statements, and most of them are unlikely to have ethical issues. Therefore, please read the statements and use your best judgment to determine whether the paper should be flagged. Please don’t automatically flag a paper simply because it includes an ethics/impact statement.
- Regardless of whether you flag a paper, you should review the paper carefully and answer the rest of the questions on the review form.
Anonymity requirement: Please indicate whether you know the identity of some authors of this paper. If yes, please provide more details (e.g., the url of the version posted online) in the “Author identify justification” box.
Overall recommendation: Here you are asked to synthesize the above and come up with your own recommendation for the paper.
- Like ACL 2020, we have used a 5-point scale with a half point increments. The detailed explanation for each point level is provided in the review form. These numbers are just a concise way of expressing your overall opinion and relative importance of the factors mentioned above.
- Different from ACL 2020, we are allowing a rating of 3 (ambivalent). Please try to take a stand on whether the paper is above or below the borderline, e.g., by selecting 2.5 or 3.5. However, if you think this is indeed a borderline paper or you are not able to decide, you should use 3.
- Decisions will be made not just on the scores and certainly not on average scores, but will also take into account the whole review, reviewer discussion and area chair meta-reviews and recommendations. However, it is important to align your recommendation with the reasoning given above, so that authors will be able to understand the motivation for the recommendations and how decisions were arrived at.
Reviewer confidence: This section should be used to inform the committee and authors how confident you are about your recommendation, taking into account your own expertise and familiarity with this area and the paper’s contents.
Recommendations for awards: For Best Paper Award, please be open minded and feel free to nominate good quality papers even though they may not be the typical types. These can be a survey paper, an opinion paper, a paper about resources and datasets, a paper for low resource language, an analysis paper, etc. A committee will evaluate best paper candidates, and we would like to have a wide variety of paper types in the candidate pool, not just vanilla empirical research papers.
Paper focus (for Theme paper only): If the paper is a theme paper, please check all the topic(s) covered by the paper.

2. Questions and additional feedback for the authors: Since we will have an author response process, for questions you would like the author(s) to respond to during the response period, please include them here. This is also the place for you to give suggestions to the authors to help them improve the paper for the final version (or a future submission).

3. Changes after the rebuttal period: There will be an author response period. It is important for you to check whether author responses have cleared up your questions. This may influence your overall recommendation and the core review. If that’s the case, please update your recommendation and review in Sections 1 and 2 accordingly, and briefly explain your decision for changing or keeping your original overall recommendation in Section 3.

4. Confidential information to the committee and chairs: Your comments in this section will not be shared with the authors. There are two boxes in this section. The first one will be shared with your peer reviewers and the area chair, and the second one will be shared with the area chair, senior area chairs, and program chairs.

Supplementary Materials

Supplementary materials are allowed as a stand-alone document uploaded as additional files. Supplementary materials are, as the name suggests, supplementary, and you have no obligation to read them. You should treat them like other citations in submissions that may be helpful in understanding background or details beyond the scope of the paper itself.

However, as noted above, given the requirement for reproducibility, authors may provide additional information about their datasets and experiments in Appendix, and attach a zip file with resources such as code and data. Please take some time to check those, if applicable.

Secondary Reviewers

As in most previous NLP conferences, you are allowed to solicit help from others. However, when it comes to writing the final review and giving the final scores, we expect you to take the secondary reviewer’s review, revise it and adjust the scores when you see fit. Essentially, the final review should reflect your own opinions about the paper, and you need to be able to justify the opinions you present in the final review.

When you send the secondary reviewer any material related to the submission (e.g., the paper itself, the supplementary materials), please remind him/her to keep the material confidential.

Format of Submissions

The program chairs, the senior area chairs, and the area chairs have identified some submissions that violate our formatting guidelines and have desk-rejected those submissions. However, it is possible that some violations were not detected in the first pass. Therefore, if you think the paper violates the format guidelines, please mention that on the review form, and highlight that in Section 4.

General Guidelines for ACL Reviewing

Please take a balanced approach when reviewing the papers. On the one hand, we would like to have a solid technical program with high-quality papers describing a complete piece of work. On the other hand, we also want a broad and interesting program, so please keep an open mind when evaluating and recommending the papers assigned to you.

Please note that a short paper is not a shortened long paper. Instead short papers should have a point that can be made in a few pages and present a focused contribution.

Following ACL 2020, we suggest that you consult some of the excellent advice from experienced reviewers and conference organizers on the web. In particular:

Read the excellent blog post by NLP veterans Mirella Lapata, Marco Baroni, Yoav Artzi, Emily Bender, Joel Tetreault, Ani Nenkova, and Tim Baldwin, who compiled their reviewing recommendations for ACL 2017: https://acl2017.wordpress.com/2017/02/23/last-minute-reviewing-advice/
Nikas Elmqvist from the University of Maryland has great advice on mistakes to avoid when reviewing: https://sites.umiacs.umd.edu/elm/2016/02/01/mistakes-reviewers-make/
Sample reviews by the NAACL 2018 chairs (note they have a different form): https://naacl2018.wordpress.com/2018/01/20/a-review-form-faq/
And on the lighter side: https://naacl2018.wordpress.com/2017/12/19/some-holiday-reviewing-advice/

Reviewer Mentoring Program

This year, we have a Reviewer Mentoring Program. The program is organized by the mentoring committee:
- Jing Huang, JD AI Research, China
- Antoine Bosselut, Stanford University, USA
- Christophe Gravier, Universite de Saint-Etienne/Lyon, France
The committee has made videos specifically for ACL-IJCNLP 2021, and all the reviewers are welcome to watch the videos before starting reviewing. The videos and other information will be available at:
- https://2021.aclweb.org/blog/reviewer-mentoring-program/.
For the “first-time reviewers” who have signed up for one-on-one mentoring, the mentoring committee will contact you directly with more information.

If you have any questions about your assignments, you should contact your senior area chairs directly. You should have already received an email from them via the START system.

Reviewer Mentoring Program

2021-01-27T00:00:00-06:00

Signing up to mentoring program

First-time reviewers can signup for one-to-one mentoring on a voluntarily basis at : This Link

Mentoring videos for reviewing

Several videos were created in order to help all reviewers in the reviewing process, especially first-time reviewers. The set of video includes the presentation of the mentoring program, a general reviewing tutorial, information about the review form used this year, how to consider ethics in submissions, and finally a set of guidelines on reviewing reproducibility. All the videos are accessible here and should be accessible worldwide.

Rationale for the reviewer mentoring program

Given the rapid growth of NLP in terms of number of papers and new students, it is very important for our community to mentor and train new reviewers. Pursuant to this goal, ACL 2020 launched a pilot “reviewer mentoring program” which called for each Area Chair (AC) to mentor at least one first-time ACL reviewer. Ultimately, the goal was to provide mentoring to new reviewers entering the field to ensure lasting review quality as the growth of the field continued.

ACL-IJCNLP’21 will continue and evolve this program. First-time reviewers will continue to be mentored by an area chair (AC). They will also be asked to go through several training resources. First-time reviewers will be mandatorily enrolled to complete these training resources. After that, they will be paired on a voluntary basis with an AC to be their mentor. To be mentored means that the first-time reviewer will have to submit one review (picked by SACs and the mentor together) two weeks before the standard review deadline to allow discussions with the mentor. This evolution of the reviewer mentoring program corresponds to the recommendation of the pilot experience at ACL’20.

Who is considered a first-time reviewer?

On the ACL-IJCNLP 2021 reviewer form, reviewers were asked to declare whether this is the first time they are reviewing for an *ACL conference. All of those who answered yes to that question will be considered first-time reviewers. Note this pool of reviewers includes ones who have reviewed for other conferences, because conferences can differ with respect to their review criteria and expectations. For ACL-IJCNLP’21, the first-time reviewer pool consists of 252 first-time reviewers spread among all tracks.

As a first-time reviewer, what am I supposed to do?

First of all, we thank you for volunteering to review for ACL-IJCNLP 2021. The quality of reviews directly impacts the quality of the conference and your efforts will ensure the quality of the final program. The reviewer mentoring program is here to help first-time reviewers in that regard. The mentoring program has three stages:

Stage 1 (Feb 1-12, 2021): Watching videos and optionally signing up for one-on-one mentoring program

(a) The full-paper submission deadline is Feb 1, and the review period starts around Feb 25, 2021.

(b) The mentoring committee will release reviewer tutorial videos and other optional reviewing materials around Feb 1.

(d) If after watching the videos and going over the optional materials, you want to be paired with a mentor to further improve your reviewing skills, you can sign up for Stage 2 of the program (aka the one-on-one mentoring). The deadline for signup is Feb 12. Note that the participation in Stage 2 is on a voluntary basis.

(e) The mentoring committee will then pair up the reviewers with mentors before the review period starts on Feb 25.

Stage 2 (Feb 25 - March 20): One-on-one mentoring during the review period

This step is on a voluntary basis. If you participate in this stage, you need to complete one of your reviews by March 10, ten days before the normal review due date (Your mentor will tell you beforehand which review needs to be submitted early). Your mentor will read your review and provide feedback. You can then revise your review based on the feedback. Your mentor will read the revision and make further suggestions if needed.

Stage 3 (After March 20, 2021): Mentoring program quality check

After March 20, all the first-time reviewers and mentors will be asked to complete an online survey to evaluate the effectiveness of the mentoring program. The aggregated results from the survey, with all the private information removed, will be made available to the ACL community.

As a first-time reviewer, how will I be contacted by the mentoring committee and by my mentor?

The mentoring committee will contact you using emails you provided on the reviewer form. The first email will arrive around Feb 1. Your mentor will use the same emails for communication.

As a first-time reviewer, what is expected from my review from my mentor?

Hopefully, the tutoring materials prepared by the mentoring committee will help you to become more familiar with reviewing for ACL-IJCNLP 2021. Nonetheless, those tutorial materials do not necessarily answer questions specific to a particular paper submission, such as “this paper does not do X, should I lower my overall recommendation score because of that?”. You can ask this kind of question to your mentor, besides your review.

In addition, for common questions such as “What is usually the expected time I should spend on reviewing a short paper?”, “Should I read source code provided by the authors?” The mentoring committee conducted a survey in December 2020 with Senior Area Chair and Area Chairs. Consequently, we are strongly invited to have a look at the results of that survey in its dedicated blog post page.

The mentoring program would be impossible without you and mentors

We deeply thank all the first-time reviewers for completing the program and all the ACL-IJCNLP’21 Area Chairs who will be enrolled in this program as mentors. We recognize this as an extra effort asked of them for the sake of quality of the ACL-IJCNLP’21 edition. Given the importance of review quality and the significant increase of paper submissions, we recognize this effort and we are grateful for their work in this matter.

Reviewer Mentoring Program Survey

2021-01-20T00:00:00-06:00

Rationale for the reviewer mentoring program survey

ACL-IJCNLP’21 will provide a mentoring program service to first-time reviewers. We are about to publish detailed information on this blog in the upcoming few days. In today’s blog post we want to share with the community a survey we conducted towards Senior Area Chairs and Area Chairs. During our mentoring program preparations, we asked SACs and ACs what they expect of reviews, including impact of code, appendices and data, the expected reviewing time per paper, what they would want to see in a reviewing tutorial video towards first-time reviewers, what they expect from the rebuttal phase.

Because some of these questions and answers are beyond the sole scope of the reviewer mentoring program, and probably of interest to all colleagues, we want to publicly share here the main results of that survey.

We first want to thank all 130 Senior Area Chairs and Area Chairs (24 SAC and 104 ACs) who took the time to answer this survey. A couple of questions were open questions and are not included in the statistical feedback here. We thank our SACs and ACs for this: many submitted many helpful comments in answer to the open questions “how to evaluate when there are enough experiments for leaning towards acceptance?” and “What should such a video tutorial(s) cover besides what is mentioned above? (which important aspect we missed?)”). Your input was carefully read, and it will influence how we proceed this year.

On multiple choice questions, we first asked SACs and ACs to rank some ideas we had in mind to be included in a general reviewing tutorial video for first-time reviewers. The question was as follows: “We plan to make video tutorial(s) for new reviewers (PhD 2nd, 3rd years mostly). What should be the video tutorial priorities among the following topics? Sort from most important (top) to least important (bottom).”

We then asked them how much time a reviewer should spend on average reviewing a short and a long paper respectively (If you are a first-time reviewer, please consider the upper bound obviously).

For short papers:

For long papers:

Then, we asked them how they feel about supplementary materials, mainly appendices, source code and data.

“Do you feel that reviewers for a conference like ACL/EMNLP/COLING should check *appendices*?”

“Do you feel that reviewers for a conference like ACL/EMNLP/COLING should check *code*?”

“Do you feel that reviewers for a conference like ACL/EMNLP/COLING should check *data*?”

In the case of source code, we wanted to evaluate how important is the source code in making their decision as ACs and SACs, depending on the main paper categories we could think of. “When do you think the availability of the source code is needed for you to make a decision?”

Next, a binary question shows that ACs and SACs prefer that the rebuttal allows reviewers to ultimately take a clear stance supported by arguments, rather than absolutely trying to reach a consensus. “What do you *mostly* expect at the rebuttal stage?

Reviewers discussing and ultimately resolve any conflicts (I would rather avoid review scores discrepancies)
Reviewers taking a clear stance and provide arguments (I don’t mind managing different scores as long as I have supporting arguments)

Finally, we asked them how they perceive the “harshness” of first-time reviewers on average, with respect to seasoned reviewers. This one should be taken with a grain of salt since SACs and ACs are the seasoned reviewers :-)

“From your experience, do first-time reviewers:

Tends to be more harsh than seasoned reviewers
Tends to be equally harsh or permissive than seasoned reviewers
Tends to be more permissive than seasoned reviewers”

We hope that you found these statistics useful as well, as SACs, ACs, reviewers or authors!

Stay connected, we will provide the mentoring program description on the ACL-IJCNLP blog very soon.