Reviewer Mentoring Program

Signing up to mentoring program

First-time reviewers can signup for one-to-one mentoring on a voluntarily basis at : This Link

Mentoring videos for reviewing

Several videos were created in order to help all reviewers in the reviewing process, especially first-time reviewers. The set of video includes the presentation of the mentoring program, a general reviewing tutorial, information about the review form used this year, how to consider ethics in submissions, and finally a set of guidelines on reviewing reproducibility. All the videos are accessible here and should be accessible worldwide.

Rationale for the reviewer mentoring program

Given the rapid growth of NLP in terms of number of papers and new students, it is very important for our community to mentor and train new reviewers. Pursuant to this goal, ACL 2020 launched a pilot “reviewer mentoring program” which called for each Area Chair (AC) to mentor at least one first-time ACL reviewer. Ultimately, the goal was to provide mentoring to new reviewers entering the field to ensure lasting review quality as the growth of the field continued.

ACL-IJCNLP’21 will continue and evolve this program. First-time reviewers will continue to be mentored by an area chair (AC). They will also be asked to go through several training resources. First-time reviewers will be mandatorily enrolled to complete these training resources. After that, they will be paired on a voluntary basis with an AC to be their mentor. To be mentored means that the first-time reviewer will have to submit one review (picked by SACs and the mentor together) two weeks before the standard review deadline to allow discussions with the mentor. This evolution of the reviewer mentoring program corresponds to the recommendation of the pilot experience at ACL’20.

Who is considered a first-time reviewer?

On the ACL-IJCNLP 2021 reviewer form, reviewers were asked to declare whether this is the first time they are reviewing for an *ACL conference. All of those who answered yes to that question will be considered first-time reviewers. Note this pool of reviewers includes ones who have reviewed for other conferences, because conferences can differ with respect to their review criteria and expectations. For ACL-IJCNLP’21, the first-time reviewer pool consists of 252 first-time reviewers spread among all tracks.

As a first-time reviewer, what am I supposed to do?

First of all, we thank you for volunteering to review for ACL-IJCNLP 2021. The quality of reviews directly impacts the quality of the conference and your efforts will ensure the quality of the final program. The reviewer mentoring program is here to help first-time reviewers in that regard. The mentoring program has three stages:

Stage 1 (Feb 1-12, 2021): Watching videos and optionally signing up for one-on-one mentoring program

(a) The full-paper submission deadline is Feb 1, and the review period starts around Feb 25, 2021.

(b) The mentoring committee will release reviewer tutorial videos and other optional reviewing materials around Feb 1.

(c) All the first-time reviewers are required to watch the videos during Stage 1.

(d) If after watching the videos and going over the optional materials, you want to be paired with a mentor to further improve your reviewing skills, you can sign up for Stage 2 of the program (aka the one-on-one mentoring). The deadline for signup is Feb 12. Note that the participation in Stage 2 is on a voluntary basis.

(e) The mentoring committee will then pair up the reviewers with mentors before the review period starts on Feb 25.

Stage 2 (Feb 25 - March 20): One-on-one mentoring during the review period

This step is on a voluntary basis. If you participate in this stage, you need to complete one of your reviews by March 10, ten days before the normal review due date (Your mentor will tell you beforehand which review needs to be submitted early). Your mentor will read your review and provide feedback. You can then revise your review based on the feedback. Your mentor will read the revision and make further suggestions if needed.

Stage 3 (After March 20, 2021): Mentoring program quality check

After March 20, all the first-time reviewers and mentors will be asked to complete an online survey to evaluate the effectiveness of the mentoring program. The aggregated results from the survey, with all the private information removed, will be made available to the ACL community.

As a first-time reviewer, how will I be contacted by the mentoring committee and by my mentor?

The mentoring committee will contact you using emails you provided on the reviewer form. The first email will arrive around Feb 1. Your mentor will use the same emails for communication.

As a first-time reviewer, what is expected from my review from my mentor?

Hopefully, the tutoring materials prepared by the mentoring committee will help you to become more familiar with reviewing for ACL-IJCNLP 2021. Nonetheless, those tutorial materials do not necessarily answer questions specific to a particular paper submission, such as “this paper does not do X, should I lower my overall recommendation score because of that?”. You can ask this kind of question to your mentor, besides your review.

In addition, for common questions such as “What is usually the expected time I should spend on reviewing a short paper?”, “Should I read source code provided by the authors?” The mentoring committee conducted a survey in December 2020 with Senior Area Chair and Area Chairs. Consequently, we are strongly invited to have a look at the results of that survey in its dedicated blog post page.

The mentoring program would be impossible without you and mentors

We deeply thank all the first-time reviewers for completing the program and all the ACL-IJCNLP’21 Area Chairs who will be enrolled in this program as mentors. We recognize this as an extra effort asked of them for the sake of quality of the ACL-IJCNLP’21 edition. Given the importance of review quality and the significant increase of paper submissions, we recognize this effort and we are grateful for their work in this matter.