Please use this identifier to cite or link to this item: doi:10.22028/D291-29404
Volltext verfügbar? / Dokumentlieferung
Title: Multimodal Dialogue System Evaluation : A Case Study Applying Usability Standards
Author(s): Malchanau, Andrei
Petukhova, Volha
Bunt, Harry
Editor(s): D'Haro, Luis Fernando
Banchs, Rafael E.
Li, Haizhou
Language: English
Title: Proceedings Ninth International Workshop on Spoken Dialogue Systems Technology
Startpage: 145
Endpage: 159
Publisher/Platform: Springer
Year of Publication: 2019
Place of publication: Singapore
Title of the Conference: IWSDS 2019
Place of the conference: Siracusa, Italy
Publikation type: Conference Paper
Abstract: This paper presents an approach to the evaluation of multimodal dialogue systems, applying usability metrics defined in ISO standards. Users’ perceptions of effectiveness, efficiency and satisfaction were correlated with various performance metrics derived from system logfiles and reference annotations. Usability experts rated questions from a preliminary 110-items questionnaire, and an assessment of their agreement on usability concepts has led to a selection of eight main factors: task completion and quality, robustness, learnability, flexibility, likeability, ease of use and usefulness (value) of an application. Based on these factors, an internally consistent and reliable questionnaire with 32 items (Cronbach’s alpha of 0.87) was produced. This questionnaire was used to evaluate the Virtual Negotiation Coaching system for metacognitive skills training in a multi-issue bargaining setting. The observed correlations between usability perception and derived performance metrics suggest that the overall system usability is determined by the quality of agreements reached, by the robustness and flexibility of the interaction, and by the quality of system responses.
DOI of the first publication: 10.1007/978-981-13-9443-0_13
URL of the first publication:
Link to this record: hdl:20.500.11880/28382
ISBN: 978-981-13-9442-3
Date of registration: 27-Nov-2019
Faculty: P - Philosophische Fakultät
Department: P - Sprachwissenschaft und Sprachtechnologie
Professorship: P - Prof. Dr. Dietrich Klakow
Collections:SciDok - Der Wissenschaftsserver der Universität des Saarlandes

Files for this record:
There are no files associated with this item.

Items in SciDok are protected by copyright, with all rights reserved, unless otherwise indicated.