Ethics Review Questions

We ask that reviewers consider the following questions before answering the portions of the review form concerning ethical considerations. We encourage authors to think through these questions and address them (as appropriate) in their submitted papers. Additional advice to authors is available in the ACL 2021 Ethics FAQ.

Please note that this is not an exhaustive list of types of ethical issues that can arise, but rather a set of questions meant to help researchers think through possible adverse impacts. Reviewers who spot concerns unrelated to the questions in this list (e.g., experiments whose energy consumption would have significant environmental impact, or a potentially unethical protocol for human subjects research) are encouraged to note them in the review form as well.


For papers presenting new datasets:

  • Does the paper describe how intellectual property (copyright, etc) was respected in the data collection process?
  • Does the paper describe how participants’ privacy rights were respected in the data collection process?
  • Does the paper describe how crowd workers or other annotators were fairly compensated and how the compensation was determined to be fair?
  • Does the paper indicate that the data collection process was subjected to any necessary review by an appropriate review board?


For papers presenting experiments on new OR existing datasets:

  • Does the paper describe the characteristics of the dataset in enough detail for a reader to understand which populations the technology could be expected to work for?
  • Do the claims in the paper match the experimental results, in terms of how far the results can be expected to generalize?
  • Does the paper describe the steps taken to evaluate the quality of the dataset?

    For papers concerning technology that could be deployed in user-facing applications beyond the researchers:
  • Does the paper describe how the technology would be deployed in actual use cases?
  • Does the task carried out by the technology match how it would be deployed?
  • Does the paper address possible harms when the technology is being used as intended and functioning correctly?
  • Does the paper address possible harms when the technology is being used as intended but gives incorrect results?
  • Does the paper address possible harms following from potential misuse of the technology?
  • If the system learns from user input once deployed, does the paper describe checks and limitations to the learning process?
  • Does the paper ensure that the harms identified are not likely to fall disproportionately on populations that already experience marginalization or are otherwise vulnerable?


For papers using identity characteristics (e.g., gender, race, ethnicity) as variables:

  • Does the paper use self-identifications (rather than automatically inferring or attributing identity characteristics to participants without their participation)?
  • Does the paper motivate the range of values or categories used for identity characteristics in terms of how they relate to the research question?
  • Does the paper discuss the ethical implications of categorizing people, either in training datasets or in the deployment of the technology?

Most of the text above comes from the NAACL 2021 Ethics Review Questions; we thank NAACL 2021 Ethics Chairs, Emily M. Bender and Karen Fort for drafting the prior version.