Comparing automated text classification methods

Jochen Hartmann, Juliana Huppertz, Christina Schamp and Mark Heitmann

International Journal of Research in Marketing, 2019, vol. 36, issue 1, 20-38

Abstract: Online social media drive the growth of unstructured text data. Many marketing applications require structuring this data at scales non-accessible to human coding, e.g., to detect communication shifts in sentiment or other researcher-defined content categories. Several methods have been proposed to automatically classify unstructured text. This paper compares the performance of ten such approaches (five lexicon-based, five machine learning algorithms) across 41 social media datasets covering major social media platforms, various sample sizes, and languages. So far, marketing research relies predominantly on support vector machines (SVM) and Linguistic Inquiry and Word Count (LIWC). Across all tasks we study, either random forest (RF) or naive Bayes (NB) performs best in terms of correctly uncovering human intuition. In particular, RF exhibits consistently high performance for three-class sentiment, NB for small samples sizes. SVM never outperform the remaining methods. All lexicon-based approaches, LIWC in particular, perform poorly compared with machine learning. In some applications, accuracies only slightly exceed chance. Since additional considerations of text classification choice are also in favor of NB and RF, our results suggest that marketing research can benefit from considering these alternatives.

Keywords: Text classification; Social media; Machine learning; User-generated content; Sentiment analysis; Natural language processing (search for similar items in EconPapers)
Date: 2019
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (35)

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0167811618300545
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:ijrema:v:36:y:2019:i:1:p:20-38

DOI: 10.1016/j.ijresmar.2018.09.009

Access Statistics for this article

International Journal of Research in Marketing is currently edited by Roland Rust

More articles in International Journal of Research in Marketing from Elsevier
Bibliographic data for series maintained by Catherine Liu ().