Real-time emotion detection by quantitative facial motion analysis

Jordan R Saadon, Fan Yang, Ryan Burgert, Selma Mohammad, Theresa Gammel, Michael Sepe, Miriam Rafailovich, Charles B Mikell, Pawel Polak and Sima Mofakham

PLOS ONE, 2023, vol. 18, issue 3, 1-14

Abstract: Background: Research into mood and emotion has often depended on slow and subjective self-report, highlighting a need for rapid, accurate, and objective assessment tools. Methods: To address this gap, we developed a method using digital image speckle correlation (DISC), which tracks subtle changes in facial expressions invisible to the naked eye, to assess emotions in real-time. We presented ten participants with visual stimuli triggering neutral, happy, and sad emotions and quantified their associated facial responses via detailed DISC analysis. Results: We identified key alterations in facial expression (facial maps) that reliably signal changes in mood state across all individuals based on these data. Furthermore, principal component analysis of these facial maps identified regions associated with happy and sad emotions. Compared with commercial deep learning solutions that use individual images to detect facial expressions and classify emotions, such as Amazon Rekognition, our DISC-based classifiers utilize frame-to-frame changes. Our data show that DISC-based classifiers deliver substantially better predictions, and they are inherently free of racial or gender bias. Limitations: Our sample size was limited, and participants were aware their faces were recorded on video. Despite this, our results remained consistent across individuals. Conclusions: We demonstrate that DISC-based facial analysis can be used to reliably identify an individual’s emotion and may provide a robust and economic modality for real-time, noninvasive clinical monitoring in the future.

Date: 2023
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0282730 (text/html)
https://journals.plos.org/plosone/article/file?id= ... 82730&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pone00:0282730

DOI: 10.1371/journal.pone.0282730

Access Statistics for this article

More articles in PLOS ONE from Public Library of Science
Bibliographic data for series maintained by plosone ().