This is an old revision of the document!
Table of Contents
Annotation Guide
Introduction
Our study aims at identifying stances towards ethical issues in software.
Thank you for helping us in our study. Your task is to read user posts from Reddit and label them according to a given set of codes. This guide describes the instructions that you should follow carefully to conduct the task successfully.
You should have access to a link together with this guide, pointing at the data sheets which you use to complete this task.
In the near future, I will also add a link to anonymized samples of our annotations.
Labeling posts
The task is to read the posts in the Excel sheet one by one. Then, for each Reddit post, we need to:
1
Read the Reddit post closely, both title and body. Decide if the post addresses an ethical issue in software (EIS). You may have a look at the EIS classification section to make that decision. If it does not, you label it with FALSE in the ‘has_eis’ column and move on to the next post. If it does, label it with TRUE and continue to the next step.
2
Decide which type of ethical issue the post addresses. Please see the full list of ethical issues in software with examples here and in the sidebar. You can select one or more types in the dropdown list of column ‘eis_types’. If none apply, select other.
3
Consider the perspective of the author for the last step. Which side of the issue would they take? If there is not enough information to make this decision, label it as ‘neutral’ in the stance column. Otherwise decide if the author takes side with those that see an ethical issue in software (against stance towards EIS), or if they are on the opposite side of the issue (in favor stance towards EIS). See more details and examples in the Classification section.
+
If you chose either in favor or against, try to select and copy/paste the part of the text that contributed most to your decision into the deciding_factor column.
Short Glossary
- Ethics
- Ethical issue in Software
- Reddit
- Software
- Stance
Ethics
Ethics is defined as a set of moral principles. They are the principles of conduct that govern an individual or a group. (MerriamWebster)
Therefore, ethics also define people's judgments about what is right and what is wrong.