Bitte benutzen Sie diese Referenz, um auf diese Ressource zu verweisen:
Volltext verfügbar? / Dokumentlieferung
doi:10.22028/D291-38835
Dateien zu diesem Datensatz:
Es gibt keine Dateien zu dieser Ressource.
Titel: | Establishing Annotation Quality in Multi-Label Annotations |
VerfasserIn: | Marchal, Marian Scholman, Merel Cleo Johanna ![]() Yung, Frances Pikyu ![]() Demberg, Vera ![]() |
HerausgeberIn: | Scherrer, Yves |
Sprache: | Englisch |
In: | |
Titel: | Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2022) - the 29th International Conference on Computational Linguistics : October 12-17, 2022, Gyeongju, Republic of Korea |
Seiten: | 3659-3668 |
Verlag/Plattform: | ACL |
Erscheinungsjahr: | 2022 |
Erscheinungsort: | [Stroudsburg, PA] |
Konferenzort: | Gyeongju, Republic of Korea |
DDC-Sachgruppe: | 400 Sprache, Linguistik |
Dokumenttyp: | Konferenzbeitrag (in einem Konferenzband / InProceedings erschienener Beitrag) |
Abstract: | In many linguistic fields requiring annotated data, multiple interpretations of a single item are possible. Multi-label annotations more accurately reflect this possibility. However, allowing for multi-label annotations also affects the chance that two coders agree with each other. Calculating inter-coder agreement for multi-label datasets is therefore not trivial. In the current contribution, we evaluate different metrics for calculating agreement on multi-label annotations: agreement on the intersection of annotated labels, an augmented version of Cohen’s Kappa, and precision, recall and F1. We propose a bootstrapping method to obtain chance agreement for each measure, which allows us to obtain an adjusted agreement coefficient that is more interpretable. We demonstrate how various measures affect estimates of agreement on simulated datasets and present a case study of discourse relation annotations. We also show how the proportion of double labels, and the entropy of the label distribution, influences the measures outlined above and how a bootstrapped adjusted agreement can make agreement measures more comparable across datasets in multi-label scenarios. |
Link zu diesem Datensatz: | urn:nbn:de:bsz:291--ds-388358 hdl:20.500.11880/35019 http://dx.doi.org/10.22028/D291-38835 |
Datum des Eintrags: | 26-Jan-2023 |
Fakultät: | MI - Fakultät für Mathematik und Informatik |
Fachrichtung: | MI - Informatik |
Professur: | MI - Prof. Dr. Vera Demberg |
Sammlung: | SciDok - Der Wissenschaftsserver der Universität des Saarlandes |
Alle Ressourcen in diesem Repository sind urheberrechtlich geschützt.