Efficient topic modeling for large-scale community question and answer websites

Sneh Prabha () and Neetu Sardana ()
Additional contact information
Sneh Prabha: Jaypee Institute of Information Technology
Neetu Sardana: Jaypee Institute of Information Technology

International Journal of System Assurance Engineering and Management, 2025, vol. 16, issue 2, No 15, 685-710

Abstract: Abstract Community Question and Answering (Q&A) websites hold a wealth of unstructured text. Analyzing this text using topic modeling can offer valuable insights into recent trends and technology within these communities. However, current topic modeling methods have limitations, as they often require default parameter tuning and struggle to handle large datasets effectively. We introduce a new topic modeling technique called LEFT (LDA Entropy TFIDF-based Fuzzy Modeling) to address these challenges. We evaluated LEFT using twenty-eight datasets from the Stack Exchange websites, including Artificial Intelligence (AI), Software Engineering (SE), Data Science (DS), Information Security (IS), and Quantum Computing (QC). In our study, we compared the performance of LEFT with two state-of-the-art techniques: Latent Dirichlet Allocation (LDA) (Blei et al., J Mach Learn Res 3:993–1022, 2003) and Fuzzy Latent Semantic Analysis (FLSA) (Karami et al., Int J Fuzzy Syst 20:1334–1345, 2018). Our findings indicate that LEFT outperforms existing techniques. It shows a 13.71% and 22.26% improvement in Silhouette score and CH score for large-scale datasets compared to the LDA model. Moreover, LEFT demonstrates significant improvements over FLSA and delivers comparable results to the LDA model for small-scale datasets.

Keywords: TF-IDF; Topic model; Fuzzy approach; LDA; Entropy; Community q&a websites (search for similar items in EconPapers)
Date: 2025
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s13198-024-02641-z Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:ijsaem:v:16:y:2025:i:2:d:10.1007_s13198-024-02641-z

Ordering information: This journal article can be ordered from
http://www.springer.com/engineering/journal/13198

DOI: 10.1007/s13198-024-02641-z

Access Statistics for this article

International Journal of System Assurance Engineering and Management is currently edited by P.K. Kapur, A.K. Verma and U. Kumar

More articles in International Journal of System Assurance Engineering and Management from Springer, The Society for Reliability, Engineering Quality and Operations Management (SREQOM),India, and Division of Operation and Maintenance, Lulea University of Technology, Sweden
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().