A Theory-Based Explainable Deep Learning Architecture for Music Emotion

Hortense Fong (), Vineet Kumar () and K. Sudhir ()
Additional contact information
Hortense Fong: Marketing, Columbia Business School, New York, New York 10027
Vineet Kumar: Yale School of Management, New Haven, Connecticut 06511
K. Sudhir: Yale School of Management, New Haven, Connecticut 06511

Marketing Science, 2025, vol. 44, issue 1, 196-219

Abstract: This paper develops a theory-based, explainable deep learning convolutional neural network (CNN) classifier to predict the time-varying emotional response to music. We design novel CNN filters that leverage the frequency harmonics structure from acoustic physics known to impact the perception of musical features. Our theory-based model is more parsimonious, but it provides comparable predictive performance with atheoretical deep learning models while performing better than models using handcrafted features. Our model can be complemented with handcrafted features, but the performance improvement is marginal. Importantly, the harmonics-based structure placed on the CNN filters provides better explainability for how the model predicts emotional response (valence and arousal) because emotion is closely related to consonance—a perceptual feature defined by the alignment of harmonics. Finally, we illustrate the utility of our model with an application involving digital advertising. Motivated by YouTube’s midroll ads, we conduct a laboratory experiment in which we exogenously insert ads at different times within videos. We find that ads placed in emotionally similar contexts increase ad engagement (lower skip rates and higher brand recall rates). Ad insertion based on emotional similarity metrics predicted by our theory-based, explainable model produces comparable or better engagement relative to atheoretical models.

Keywords: audio data; deep learning; explainable and interpretable AI; emotion; digital advertising; music theory (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/mksc.2022.0323 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormksc:v:44:y:2025:i:1:p:196-219

Access Statistics for this article

More articles in Marketing Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().