Interpretable Multimodal Emotion Recognition in Counseling Dialogues via Factor Analysis and Gaussian Mixture Modeling

Keita Kiuchi
Additional contact information
Keita Kiuchi: National Institute of Occupational Safety and Health, Japan

No 2udp8_v1, OSF Preprints from Center for Open Science

Abstract: Multimodal emotion recognition, integrating facial expressions and vocal features, is key to advancing human-computer interaction and mental healthcare. However, current deep learning models often lack interpretability, limiting their real-world applicability. In this study, we present a framework for emotion recognition (happy, sad, angry, neutral) that uncovers meaningful multimodal patterns. We collected online dialogue data from 99 participants, extracting facial features (OpenFace), acoustic descriptors (openSMILE), and deep audio embeddings (VGGish). Factor analysis (FA) was applied independently to each modality for dimensionality reduction, and Gaussian mixture modeling (GMM) on the combined factor scores revealed latent multimodal expression clusters. These cluster probabilities, along with participant covariates (e.g., Big Five traits; PHQ-9; GAD-7/SCAS), served as inputs to various classifiers, including XGBoost and random forest. SHAP analysis confirmed the interpretability of the clusters, illustrating how individual differences and covariates influenced emotion predictions. We identified 15 distinct expression clusters (e.g., “social smiles,” “inexpressive states,” “wry grins”), offering nuanced insights into affective displays. Although overall accuracy was modest—due to individual variability and label noise—the framework effectively highlights interpretable, fine-grained expressive patterns. This approach lays groundwork for transparent affective computing systems, such as empathetic conversational agents, emphasizing the importance of explainability in emotion-based applications.

Date: 2025-08-24
New Economics Papers: this item is included in nep-dcm
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/68aaffed8281f435f4bcbc79/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:2udp8_v1

DOI: 10.31219/osf.io/2udp8_v1

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().