Improvement of Negative Emotion Recognition in Visible Images Enhanced by Thermal Imaging

Ji-Min Lee, Young-Eun An, EunSang Bak () and Sungbum Pan ()
Additional contact information
Ji-Min Lee: SW Convergence Education Institute, Chosun University, 309 Pilmun-daero, Dong-gu, Gwang-Ju 61452, Republic of Korea
Young-Eun An: IT Research Institute, Chosun University, 309 Pilmun-daero, Dong-gu, Gwang-Ju 61452, Republic of Korea
EunSang Bak: IT Research Institute, Chosun University, 309 Pilmun-daero, Dong-gu, Gwang-Ju 61452, Republic of Korea
Sungbum Pan: IT Research Institute, Chosun University, 309 Pilmun-daero, Dong-gu, Gwang-Ju 61452, Republic of Korea

Sustainability, 2022, vol. 14, issue 22, 1-15

Abstract: Facial expressions help in understanding the intentions of others as they are an essential means of communication, revealing human emotions. Recently, thermal imaging has been playing a complementary role in emotion recognition and is considered an alternative to overcome the drawbacks of visible imaging. Notably, a relatively severe recognition error of fear among negative emotions frequently occurs in visible imaging. This study aims to improve the recognition performance of fear by using the visible and thermal images acquired simultaneously. When fear was not recognized in a visible image, we analyzed the causes of misrecognition. We thus found the condition of replacing the image with a thermal image. It improved emotion recognition performance by 4.54% on average, compared to the performance of using only visible images. Finally, we confirmed that the thermal image effectively compensated for the visible image’s shortcomings.

Keywords: facial emotion recognition; fusion of visible light image and thermal facial image; convolutional neural network; robustness to classify in negative emotion (search for similar items in EconPapers)
JEL-codes: O13 Q Q0 Q2 Q3 Q5 Q56 (search for similar items in EconPapers)
Date: 2022
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2071-1050/14/22/15200/pdf (application/pdf)
https://www.mdpi.com/2071-1050/14/22/15200/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jsusta:v:14:y:2022:i:22:p:15200-:d:974462

Access Statistics for this article

Sustainability is currently edited by Ms. Alexandra Wu

More articles in Sustainability from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().