DeepConnection: classifying momentary relationship state from images of romantic couples

Maximiliane Uhlich and Daniel Bojar ()

Journal of Computational Social Science, 2021, vol. 4, issue 2, No 9, 653 pages

Abstract: Abstract Detecting momentary relationship state and quality in romantic couples is an important endeavor for relationship research, couple therapy, and of course couples themselves. Yet current methods to achieve this are intrusive, asynchronous, plagued by ceiling effects, and only assess subjective responses to questionnaires while trying to capture the objective state of a relationship. According to social appraisal theory, human beings rely on emotional responses to assess interpersonal situations, a key element for relationship functioning in couples. Using couples is particularly advantageous as strong emotional reactions are triggered in romantic relationships. Here, we employ deep learning methods to assess the momentary relationship state of romantic couples from predominantly stock images via facial and bodily emotion expression and other features. Our new model, DeepConnection, comprises pre-trained residual neural networks, spatial pyramid pooling layers, and power mean transformations to extract relevant features from images for binary classification. With this, we achieved an average accuracy of nearly 97% on a separate validation dataset. We also engaged in model interpretation using Gradient-weighted Class Activation Mapping (Grad-CAM) to identify which features allow DeepConnection to detect binarized momentary relationship state. To demonstrate generalizability and robustness, we used DeepConnection to analyze videos of couples exhibiting a range of different postures and facial expressions. Here, we achieved an average accuracy of about 85% with a trained DeepConnection model. The work presented here could inform couples, advance relationship research, and find application in couple therapy to assist the therapist.

Keywords: Machine learning; Relationship research; Computer vision; Neural network; Emotion expression (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s42001-021-00102-2 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:jcsosc:v:4:y:2021:i:2:d:10.1007_s42001-021-00102-2

Ordering information: This journal article can be ordered from
http://www.springer. ... iences/journal/42001

DOI: 10.1007/s42001-021-00102-2

Access Statistics for this article

Journal of Computational Social Science is currently edited by Takashi Kamihigashi

More articles in Journal of Computational Social Science from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().