Semi-formal Evaluation of Conversational Characters

被引:15
作者
Artstein, Ron [1 ]
Gandhe, Sudeep [1 ]
Gerten, Jillian [1 ]
Leuski, Anton [1 ]
Traum, David [1 ]
机构
[1] Univ So Calif, Inst Creat Technol, Marina Del Rey, CA 90292 USA
来源
LANGUAGES: FROM FORMAL TO NATURAL | 2009年 / 5533卷
关键词
D O I
10.1007/978-3-642-01748-3_2
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
Conversational dialogue systems cannot be evaluated in a fully formal manner, because dialogue is heavily dependent on context and current dialogue theory is not precise enough to specify a target output ahead of time. Instead, we evaluate dialogue systems in a semi-formal manner, using human judges to rate the coherence of a conversational character and correlating these judgments with measures extracted from within the system. We present a series of three evaluations of a single conversational character over the course of a year, demonstrating how this kind of evaluation helps bring about an improvement in overall dialogue coherence.
引用
收藏
页码:22 / 35
页数:14
相关论文
共 14 条
[1]  
[Anonymous], 2007, Proc. of the 8th SIGdial workshop on Discourse and Dialogue
[2]  
[Anonymous], 1993, Comput. Linguist., DOI DOI 10.21236/ADA273556
[3]  
[Anonymous], ELRA WORKSH EV
[4]  
ARTSTEIN R, 2008, 26 ARM SCI C ORL FLO
[5]   Inter-Coder Agreement for Computational Linguistics [J].
Artstein, Ron ;
Poesio, Massimo .
COMPUTATIONAL LINGUISTICS, 2008, 34 (04) :555-596
[6]   Answering the Call for a Standard Reliability Measure for Coding Data [J].
Hayes, Andrew F. ;
Krippendorff, Klaus .
COMMUNICATION METHODS AND MEASURES, 2007, 1 (01) :77-89
[7]  
Krippendorff K., 1980, CONTENT ANAL INTRO I, P129
[8]  
Leuski A., 2006, P 7 SIGDIAL WORKSHOP, P18
[9]  
LEUSKI A, 2008, 26 ARM SCI C ORL FLO
[10]   A stochastic model of human-machine interaction for learning dialog strategies [J].
Levin, E ;
Pieraccini, R ;
Eckert, W .
IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, 2000, 8 (01) :11-23