Rank-frequency distribution of natural languages: A difference of probabilities approach

Germinal Cocho, Rosalío F. Rodríguez, Sergio Sánchez, Jorge Flores, Carlos Pineda and Carlos Gershenson

Physica A: Statistical Mechanics and its Applications, 2019, vol. 532, issue C

Abstract: In this paper we investigate the time variation of the rank k of words for six Indo-European languages using the Google Books N-gram Dataset. Based on numerical evidence, we regard k as a random variable whose dynamics may be described by a Fokker–Planck equation which we solve analytically. For low ranks the distinct languages behave differently, maybe due to the syntax rules, whereas for k>50 the law of large numbers predominates. We analyze the frequency distribution of words using the data and their adjustment in terms of time-dependent probability density distributions. We find small differences between the data and the fits due to conflicting dynamic mechanisms, but the data show a consistent behavior with our general approach. For the lower ranks the behavior of the data changes among languages presumably, again, due to distinct dynamic mechanisms. We discuss a possible origin of these differences and assess the novel features and limitations of our work.

Keywords: Rank dynamics; Languages; Master equation; Fokker–Planck equation (search for similar items in EconPapers)
Date: 2019
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0378437119310507
Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:phsmap:v:532:y:2019:i:c:s0378437119310507

DOI: 10.1016/j.physa.2019.121795

Access Statistics for this article

Physica A: Statistical Mechanics and its Applications is currently edited by K. A. Dawson, J. O. Indekeu, H.E. Stanley and C. Tsallis

More articles in Physica A: Statistical Mechanics and its Applications from Elsevier
Bibliographic data for series maintained by Catherine Liu ().