Reproducibility FAQs

Q1: As an author, am I required to full out the reproducibility checklist in the full-paper submission form?

A1: Yes. You won’t be able to submit the full paper without filling out the reproducibility checklist first. Note that some items in the checklist may not be relevant to your submission. For example, a dataset paper might have no code to be released. In that case, you choose “Not applicable” for those items on the checklist.

Q2: I have completed the checklist in the submission form. Am I required to address those items explicitly in the submission itself (i.e., in the main paper or supplementary material)?

A2: The items from the checklist are not required to be addressed explicitly in a submission, but authors may find their submissions to be of higher quality if they do so for items that are relevant to their work. Please use your best judgment to determine what checklist items are relevant to your work and what items should be addressed explicitly in the submission itself.

Q3: How are the checklist responses used during the reviewing process?

A3: The checklist is designed to remind authors of items they could address in their submissions to help the reviewers to understand and evaluate the work, similar to a pre-flight checklist or pre-surgery checklist. The checklist responses are visible to reviewers and the review form will ask reviewers to rate a submission with respect to whether the submission includes enough information for reproducibility. However, the reviewers will rate the submission based on the submission itself, not simply based on the checklist responses. Also, the checklist responses are visible to Program Chairs, who might take them into consideration when making the final accept/reject decisions.

Q4: What if addressing a checklist item in the submission would deanonymize a submission?

A4: You should address checklist items without revealing your identity, similar to the way you use anonymous for self-citation. If that’s impossible (e.g., the github link for the source code), don’t address that item in the submission. Blind reviewing should take precedence over the reproducibility checklist.

Q5: Should we submit code as part of supplementary materials?

A5: Code is recommended for camera ready. It is welcome for initial submission as long as author identities are not disclosed. We recommend checking the ML Code Completeness checklist for best practices.

Q6: Should we address checklist items in the main paper or in the supplementary materials?

A6: It depends on which place is more appropriate. For example, important values such as the number of model parameters and the size of training data should be included in the main paper, whereas less important ones can be included in the appendix. This is the same kind of decision you have to make for other types of information such as math formulas. Just remember that reviewers have access to supplementary materials, but they are not required to read them.

Q7: If our paper addresses all the items on the checklist, is that sufficient for reproducibility?

A7: Not necessarily. NLP research is quite varied and no checklist could cover all the items necessary to reproduce all papers. Use your best judgement to include items relevant for your work, regardless of whether they are in the reproducibility checklist.

Q8: As a reviewer, should I lower my rating of a submission if it doesn’t address some checklist items in the submission?

A8: Please see Q2 above. Reviewers should use their best judgment to determine whether certain checklist items should be addressed in the submission.