Stock market forecasting based on machine learning: The role of investor sentiment

Tingting Ren and Shaofang Li

Physica A: Statistical Mechanics and its Applications, 2025, vol. 666, issue C

Abstract: Stock market prediction remains a classical yet challenging problem, with the focus on the investor sentiment growing increasing significant in big data era. This analysis delves into the question whether and how predicable is the stock market when considering investor sentiment. By leveraging the initial and customized LM financial lexicon and Vader technology, Word2vec and Doc2vec and BERT embedding vector method (along with two fine-tuned models: FinBERT and SentiBERT), we first construct nine investor sentiment indexes based on the textual data from Twitter between November 2019 and December 2021. And then we employ three machine learning algorithms (SVR, AdaBoost, and RF) to predict the daily return of the S&P 500 index. The experiment results confirm that the investor sentiment index can enhance prediction accuracy beyond the market indicator, aligning with prior research. Embedding vector methods exhibit superior performance compared to the fine-tuned models, and the customized dictionaries outperform their traditional counterparts. Furthermore, the composite sentiment index, integrating all the single indexes, achieves the best overall performance. To further validate our findings, we conduct robustness checks on the DJIA index and across different economic cycles, observe that the single sentiment index performs worse with shorter datasets, whereas the composite index demonstrates consistent improvement in both volatile and steady periods. These findings offer valuable insights for future research and provide practical applications in stock market prediction.

Keywords: Investor sentiment; Stock market forecasting; Text mining; Machine learning (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0378437125001852
Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:phsmap:v:666:y:2025:i:c:s0378437125001852

DOI: 10.1016/j.physa.2025.130533

Access Statistics for this article

Physica A: Statistical Mechanics and its Applications is currently edited by K. A. Dawson, J. O. Indekeu, H.E. Stanley and C. Tsallis

More articles in Physica A: Statistical Mechanics and its Applications from Elsevier
Bibliographic data for series maintained by Catherine Liu ().