Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues

Zixiu Wu, Simone Balloccu, Vivek Kumar, Rim Helaoui, Diego Reforgiato Recupero and Daniele Riboni ()
Additional contact information
Zixiu Wu: Philips Research, High Tech Campus, 5656 AE Eindhoven, The Netherlands
Simone Balloccu: Department of Computing Science, University of Aberdeen, Aberdeen AB24 3FX, UK
Vivek Kumar: Department of Mathematics and Computer Science, University of Cagliari, 09124 Cagliari, Italy
Rim Helaoui: Philips Research, High Tech Campus, 5656 AE Eindhoven, The Netherlands
Diego Reforgiato Recupero: Department of Mathematics and Computer Science, University of Cagliari, 09124 Cagliari, Italy
Daniele Riboni: Department of Mathematics and Computer Science, University of Cagliari, 09124 Cagliari, Italy

Future Internet, 2023, vol. 15, issue 3, 1-26

Abstract: Research on the analysis of counselling conversations through natural language processing methods has seen remarkable growth in recent years. However, the potential of this field is still greatly limited by the lack of access to publicly available therapy dialogues, especially those with expert annotations, but it has been alleviated thanks to the recent release of AnnoMI, the first publicly and freely available conversation dataset of 133 faithfully transcribed and expert-annotated demonstrations of high- and low-quality motivational interviewing (MI)—an effective therapy strategy that evokes client motivation for positive change. In this work, we introduce new expert-annotated utterance attributes to AnnoMI and describe the entire data collection process in more detail, including dialogue source selection, transcription, annotation, and post-processing. Based on the expert annotations on key MI aspects, we carry out thorough analyses of AnnoMI with respect to counselling-related properties on the utterance, conversation, and corpus levels. Furthermore, we introduce utterance-level prediction tasks with potential real-world impacts and build baseline models. Finally, we examine the performance of the models on dialogues of different topics and probe the generalisability of the models to unseen topics.

Keywords: dialogue; counselling; motivational interviewing; natural language processing; dataset (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2023
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/15/3/110/pdf (application/pdf)
https://www.mdpi.com/1999-5903/15/3/110/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:15:y:2023:i:3:p:110-:d:1096880

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().