Using cross‐evaluation to evaluate interactive QA systems

Ying Sun, Paul B. Kantor and Emile L. Morse

Journal of the American Society for Information Science and Technology, 2011, vol. 62, issue 9, 1653-1665

Abstract: In this article, we report on an experiment to assess the possibility of rigorous evaluation of interactive question‐answering (QA) systems using the cross‐evaluation method. This method takes into account the effects of tasks and context, and of the users of the systems. Statistical techniques are used to remove these effects, isolating the effect of the system itself. The results show that this approach yields meaningful measurements of the impact of systems on user task performance, using a surprisingly small number of subjects and without relying on predetermined judgments of the quality, or of the relevance of materials. We conclude that the method is indeed effective for comparing end‐to‐end QA systems, and for comparing interactive systems with high efficiency.

Date: 2011
References: Add references at CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://doi.org/10.1002/asi.21585

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:jamist:v:62:y:2011:i:9:p:1653-1665

Ordering information: This journal article can be ordered from
https://doi.org/10.1002/(ISSN)1532-2890

Access Statistics for this article

More articles in Journal of the American Society for Information Science and Technology from Association for Information Science & Technology
Bibliographic data for series maintained by Wiley Content Delivery ().