Bitte benutzen Sie diese Referenz, um auf diese Ressource zu verweisen: doi:10.22028/D291-42311
Volltext verfügbar? / Dokumentlieferung
Titel: Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design
VerfasserIn: Pyatkin, Valentina
Yung, Frances Pikyu
Scholman, Merel Cleo Johanna
Tsarfaty, Reut
Dagan, Ido
Demberg, Vera
Sprache: Englisch
Titel: Transactions of the Association for Computational Linguistics
Bandnummer: 11
Seiten: 1014-1032
Verlag/Plattform: ACL
Erscheinungsjahr: 2023
DDC-Sachgruppe: 004 Informatik
400 Sprache, Linguistik
Dokumenttyp: Journalartikel / Zeitschriftenartikel
Abstract: Disagreement in natural language annotation has mostly been studied from a perspective of biases introduced by the annotators and the annotation frameworks. Here, we propose to analyze another source of bias—task design bias, which has a particularly strong impact on crowdsourced linguistic annotations where natural language is used to elicit the interpretation of lay annotators. For this purpose we look at implicit discourse relation annotation, a task that has repeatedly been shown to be difficult due to the relations’ ambiguity. We compare the annotations of 1,200 discourse relations obtained using two distinct annotation tasks and quantify the biases of both methods across four different domains. Both methods are natural language annotation tasks designed for crowdsourcing. We show that the task design can push annotators towards certain relations and that some discourse relation senses can be better elicited with one or the other annotation approach. We also conclude that this type of bias should be taken into account when training and testing models.
DOI der Erstveröffentlichung: 10.1162/tacl_a_00586
URL der Erstveröffentlichung: https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00586/117215/Design-Choices-for-Crowdsourcing-Implicit
Link zu diesem Datensatz: urn:nbn:de:bsz:291--ds-423115
hdl:20.500.11880/37983
http://dx.doi.org/10.22028/D291-42311
ISSN: 2307-387X
Datum des Eintrags: 1-Jul-2024
Fakultät: MI - Fakultät für Mathematik und Informatik
Fachrichtung: MI - Informatik
Professur: MI - Prof. Dr. Vera Demberg
Sammlung:SciDok - Der Wissenschaftsserver der Universität des Saarlandes

Dateien zu diesem Datensatz:
Es gibt keine Dateien zu dieser Ressource.


Alle Ressourcen in diesem Repository sind urheberrechtlich geschützt.