Please use this identifier to cite or link to this item: http://hdl.handle.net/1959.14/156613
26 Visitors27 Hits0 Downloads
Validating the web-based evaluation of NLG systems
Annual Meeting of the Association for Computational Linguistics (47th : 2009) and the International Joint Conference on Natural Language Processing of the AFNLP (4th : 2009) (4 August 2009 : Singapore)
ACL-IJCNLP 2009 : Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the AFNLP : proceedings of the conference short papers, p.301-304
The GIVE Challenge is a recent shared task in which NLG systems are evaluated over the Internet. In this paper, we validate this novel NLG evaluation methodology by comparing the Internet-based results with results we collected in a lab experiment. We find that the results delivered by both methods are consistent, but the Internet-based approach offers the statistical power necessary for more fine-grained evaluations and is cheaper to carry out.