A comparison of three think-aloud protocols used to evaluate a voice intelligent agent that expresses emotions

Xiang Ji and Pei-Luen Patrick Rau

Behaviour and Information Technology, 2019, vol. 38, issue 4, 375-383

Abstract: This study proposes two adapted think-aloud protocols for the evaluation of a voice intelligent agent. In the adapted retrospective think-aloud (RTA) protocol, users verbalise their thoughts based on the chat history after task-completion. In the adapted interactive think-aloud (ITA) protocol, users verbalise their thoughts regarding the intelligent agent being evaluated without the help of a facilitator. This study compares these two protocols with the classical think-aloud protocol (CTA) for evaluating an intelligent agent in terms of task time and verbal utterances. The influence of the intelligent agent’s emotional expression is also considered. The results suggest RTA is suitable for collecting user experience and causal explanation of utterances, CTA for collecting recommendation and prediction utterances, and ITA for collecting problem formulation and recommendation utterances. Furthermore, CTA and RTA can collect more total utterances, while CTA and ITA are influenced less the VIA’s emotional expression. This study provides guidelines by which future evaluators can choose suitable think-aloud protocols.

Date: 2019
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/0144929X.2018.1535621 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:tbitxx:v:38:y:2019:i:4:p:375-383

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/tbit20

DOI: 10.1080/0144929X.2018.1535621

Access Statistics for this article

Behaviour and Information Technology is currently edited by Dr Panos P Markopoulos

More articles in Behaviour and Information Technology from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().