Correlation analysis of short text based on network model

Dongyang Yan, Keping Li and Jingjing Ye

Physica A: Statistical Mechanics and its Applications, 2019, vol. 531, issue C

Abstract: Correlation of words in the text is of great importance in text analysis like text retrieval, keywords extraction, and text clustering. For short text, because of the limited information of text content, it is difficult to catch the correlation well among words. In this paper, we propose an algorithm based on the complex network to calculate the correlation of words in short texts. A new variable Edge-degree is proposed and used in studying the network model of texts. By using fluctuation analysis, we give the condition that Edge-degree correlation between words exists beyond nearest neighbors. Further analysis shows that numerical results of the fluctuation function of Edge-degree act a power law distribution and that the scaling exponent diverges at a long distance under the finite size effect and varies in different texts. The fluctuation function separates the words in a text into different clusters, and this property is used to measure inner-correlation of different words. Hub nodes act a significant influence on the long-range Edge-degree correlation through changing the linear trend of the fluctuation function in a log–log plot.

Keywords: Long-range correlation; Network model; Short text; Fluctuation analysis (search for similar items in EconPapers)
Date: 2019
References: Add references at CitEc
Citations: View citations in EconPapers (6)

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0378437119309860
Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:phsmap:v:531:y:2019:i:c:s0378437119309860

DOI: 10.1016/j.physa.2019.121728

Access Statistics for this article

Physica A: Statistical Mechanics and its Applications is currently edited by K. A. Dawson, J. O. Indekeu, H.E. Stanley and C. Tsallis

More articles in Physica A: Statistical Mechanics and its Applications from Elsevier
Bibliographic data for series maintained by Catherine Liu ().