What's in a word graph evaluation and enhancement of word lattices

Abstract

During the last few years, word graphs have been gaining increasing interest within the speech community as the primary interface between speech recognizers and language processing modules. Both development and evaluation of graph-producing speech decoders require generally accepted measures of word graph quality. While the notion of recognition accuracy can easily be extended to word graphs, a meaningful measure of word graph size has not yet surfaced. We argue, that the number of derivation steps a theoretical parser would need to process all unique sub-paths in a graph could provide a measure that is both application oriented enough to be meaningful and general enough to allow a useful comparison of word recognizers across different applications. This paper discusses various measures that are used, or could be used, to measure word graph quality. Using real-life data (word graphs evaluated in the 1996 Verbmobil acoustic evaluation), it is demonstrated how different measures can affect evaluation results. Finally, some algorithms are presented that can significantly improve word graph quality with respect to specific quality measures.