Evaluating a scheme for dialogue annotation

Abstract

This paper describes the evaluation of a coding scheme for the segmentation and annotation of transliterated spoken dialogues with dialogue act information. Using the kappa method for measuring the reliability of this coding scheme we find that both for segmentation and labeling we receive reliable results. It is shown that this observation holds both for an evaluation of intercoder reliability and for the stability of coding by one coder over time. Our studies also suggest some improvements of our coding scheme which will be incorporated into the scheme for future use.