Research Wiki
Trace:

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
eis:resources:annotation_report [2025/03/21 12:46] – tomeis:resources:annotation_report [2026/07/02 23:34] (current) – tom
Line 1: Line 1:
-<WRAP group> 
-<WRAP third column> 
-<wrap lo>Research notes.\\ 
-March 2025.</wrap> 
-</WRAP> 
-<WRAP third column> 
 ====== Report on Annotation strategy and outcomes ====== ====== Report on Annotation strategy and outcomes ======
-</WRAP> 
  
-<WRAP third column> +//Research notes March 2025.//
- +
-</WRAP> +
-</WRAP>+
  
  
Line 23: Line 13:
 The annotation uses a predefined set of labels, namely the ethical issue taxonomy established in [[https://doi.org/10.1109/ICSE-SEIS58686.2023.00013|(Olson et al. 2023)]] and extended in [[https://doi.org/10.48550/arXiv.2503.12640|(Elias et al. 2024)]]. The annotation uses a predefined set of labels, namely the ethical issue taxonomy established in [[https://doi.org/10.1109/ICSE-SEIS58686.2023.00013|(Olson et al. 2023)]] and extended in [[https://doi.org/10.48550/arXiv.2503.12640|(Elias et al. 2024)]].
 The annotation guide instructs annotators to read the post and consider if the post contains an ethical issue in software. A post is annotated if it contains a description of an issue that fits the definition of an ethical issue type. The annotator selects all types that fit, having the option to define a new type in the rare case where none of the current types fit an apparent ethical issue. Annotators are instructed to highlight the piece of the text that led to their decision. The annotation guide instructs annotators to read the post and consider if the post contains an ethical issue in software. A post is annotated if it contains a description of an issue that fits the definition of an ethical issue type. The annotator selects all types that fit, having the option to define a new type in the rare case where none of the current types fit an apparent ethical issue. Annotators are instructed to highlight the piece of the text that led to their decision.
 +
 Lastly, annotators assign the stance label, determining the Redditor's position on the issue. The **Against** stance is defined as //a position that opposes the existence or continuation of an ethical issue//, often expressed in the form of a concern. The **In favor** stance is assigned if //a person favors the continuation of a known ethical issue in software//, according to our definitions. A clear case would constitute the acknowledgement of a known issue while expressing the wish for this to continue. But this stance may also be found in cases where a person engages in, or supports, clearly unethical or illegal activity, such as purposefully and maliciously breaching platform policy. Lastly, the **Neutral** label is given if //a person does not describe their position// or if annotators cannot establish enough context to make a confident judgement. This includes the sharing of news articles without comment. Lastly, annotators assign the stance label, determining the Redditor's position on the issue. The **Against** stance is defined as //a position that opposes the existence or continuation of an ethical issue//, often expressed in the form of a concern. The **In favor** stance is assigned if //a person favors the continuation of a known ethical issue in software//, according to our definitions. A clear case would constitute the acknowledgement of a known issue while expressing the wish for this to continue. But this stance may also be found in cases where a person engages in, or supports, clearly unethical or illegal activity, such as purposefully and maliciously breaching platform policy. Lastly, the **Neutral** label is given if //a person does not describe their position// or if annotators cannot establish enough context to make a confident judgement. This includes the sharing of news articles without comment.
  
Line 38: Line 29:
 </WRAP> </WRAP>
 <WRAP half column> <WRAP half column>
-{{ :eis:resources:fulldataset.png?nolink&600 | Number of posts per software domain.}} +<wrap figure>{{ :eis:resources:fulldataset.png?nolink&600 | Number of posts per software domain.}} 
-<WRAP center round box 60%> + 
-Fig.1. Number of posts per software domain. +**Fig.1.** Number of posts per software domain. 
-</WRAP>+</wrap> 
 + 
 +<wrap figure>{{ :eis:resources:claudelabels.png?nolink&600 | Claude labels.}}
  
-{{ :eis:resources:claudelabels.png?nolink&600 | Claude labels.}} +**Fig.2.** Claude labels. 
-<WRAP center round box 60%> +</wrap>
-Fig.2. Claude labels. +
-</WRAP>+
 </WRAP> </WRAP>
 </WRAP> </WRAP>
  
 ===== Preparation ===== ===== Preparation =====
 +
 +The annotators prepared by reading the annotation guidelines, reading the definitions of the ethical issue types and 
  
 ===== Preliminary results ===== ===== Preliminary results =====
eis/resources/annotation_report.1742557586.txt.gz · Last modified: by tom