Home | Publications | HGW+24

Investigating Labeler Bias in Face Annotation for Machine Learning

MCML Authors

Luke Haliburton

Dr.

* Former Member

→ Group Albrecht Schmidt
Human-Centered Ubiquitous Media

Albrecht Schmidt

Prof. Dr.

Principal Investigator

Human-Centered Ubiquitous Media

Abstract

In a world increasingly reliant on artificial intelligence, it is more important than ever to consider the ethical implications of artificial intelligence. One key under-explored challenge is labeler bias — bias introduced by individuals who label datasets — which can create inherently biased datasets for training and subsequently lead to inaccurate or unfair decisions in healthcare, employment, education, and law enforcement. Hence, we conducted a study (N=98) to investigate and measure the existence of labeler bias using images of people from different ethnicities and sexes in a labeling task. Our results show that participants hold stereotypes that influence their decision-making process and that labeler demographics impact assigned labels. We also discuss how labeler bias influences datasets and, subsequently, the models trained on them. Overall, a high degree of transparency must be maintained throughout the entire artificial intelligence training process to identify and correct biases in the data as early as possible.

inproceedings HGW+24