Differences

This shows you the differences between two versions of the page.

--- eis:resources:annotation_report [2025/03/21 11:03] – created tom
+++ eis:resources:annotation_report [2026/07/02 23:34] (current) – tom
@@ Line 1: / Line 1: @@
 ====== Report on Annotation strategy and outcomes ======
+//Research notes March 2025.//
 ===== Method =====
+<WRAP group>
+<WRAP half column>
+We construct a manual annotation task to determine stances towards instances of ethical issues in software related Reddit posts.
+Two annotators, the author and a research assistant, annotate a dataset of 1000 posts. They follow this [[eis:resources:annotation_guide|annotation guide]], an adaptation of the annotation guide used by [[https://doi.org/10.1109/ICSE-SEIS58686.2023.00013|Olson et al.]]
+Annotators first conduct a trial annotation round to establish sufficient common understanding of the task. They annotate 100 randomly sampled posts from the full dataset of 65769 software related posts and compare their agreement before the main task. They must achieve a substantial agreement of Cohen's **kappa > 0.6** to continue, otherwise another trial is conducted. All disagreements in the trial are discussed to establish a better common understanding of the definitions of the [[eis:home|ethical issue types]].
+The annotation uses a predefined set of labels, namely the ethical issue taxonomy established in [[https://doi.org/10.1109/ICSE-SEIS58686.2023.00013|(Olson et al. 2023)]] and extended in [[https://doi.org/10.48550/arXiv.2503.12640|(Elias et al. 2024)]].
+The annotation guide instructs annotators to read the post and consider if the post contains an ethical issue in software. A post is annotated if it contains a description of an issue that fits the definition of an ethical issue type. The annotator selects all types that fit, having the option to define a new type in the rare case where none of the current types fit an apparent ethical issue. Annotators are instructed to highlight the piece of the text that led to their decision.
+Lastly, annotators assign the stance label, determining the Redditor's position on the issue. The **Against** stance is defined as //a position that opposes the existence or continuation of an ethical issue//, often expressed in the form of a concern. The **In favor** stance is assigned if //a person favors the continuation of a known ethical issue in software//, according to our definitions. A clear case would constitute the acknowledgement of a known issue while expressing the wish for this to continue. But this stance may also be found in cases where a person engages in, or supports, clearly unethical or illegal activity, such as purposefully and maliciously breaching platform policy. Lastly, the **Neutral** label is given if //a person does not describe their position// or if annotators cannot establish enough context to make a confident judgement. This includes the sharing of news articles without comment.
+The two annotators are given a stratified dataset of 1000 posts. The annotator set two checkpoints, one at a 100 posts and one at 500, to compare labels and discuss disagreement and unclarities in the guidelines.
+Throughout the process, very complex cases are further to be discussed with researcher's specialized in the field.
+==== Data ====
+After the trial sample was extracted and annotated, we decide to filter the dataset for posts with a higher likelihood to contain an ethical issue due to a very low ratio of ~20%.
+From the 65769 posts, we extract a stratified sample of 4998 posts over all software domains, ensuring that at least one post from every subreddit is included.
+We filter the posts by prompting Anthropic's Claude (claude-3-5-sonnet-latest) API for each of the 4998 posts, asking whether a post contains an ethical issue according to the given definition and set of types. The full prompt can be found in the Appendix [todo].
+Claude is instructed to answer with TRUE or FALSE, and to resort to the former when in doubt. This increases the likelihood of not missing more complex cases.
+We are aware of the limitations of this novel approach. We accept potential biases of Claude to ensure a higher density of interesting cases for analysis. This speeds up our approach to learn how to automatically assign stances towards ethical issues. Future work will establish approaches to filter occurrences of ethical issues more reliably. Works such as that of [[https://doi.org/10.1109/ICSE-SEIS58686.2023.00013|Olson et al.]] are an example of this, trying to also highlight minority cases.
+Claude assigned the TRUE label to 1756 posts, the FALSE label to 3227 posts and gibberish in 15 cases.
+</WRAP>
+<WRAP half column>
+<wrap figure>{{ :eis:resources:fulldataset.png?nolink&600 | Number of posts per software domain.}}
+**Fig.1.** Number of posts per software domain.
+</wrap>
+<wrap figure>{{ :eis:resources:claudelabels.png?nolink&600 | Claude labels.}}
+**Fig.2.** Claude labels.
+</wrap>
+</WRAP>
+</WRAP>
 ===== Preparation =====
+The annotators prepared by reading the annotation guidelines, reading the definitions of the ethical issue types and
 ===== Preliminary results =====