Randomness versus specifics for word-frequency distributions

Xiaoyong Yan and Petter Minnhagen

Physica A: Statistical Mechanics and its Applications, 2016, vol. 444, issue C, 828-837

Abstract: The text-length-dependence of real word-frequency distributions can be connected to the general properties of a random book. It is pointed out that this finding has strong implications, when deciding between two conceptually different views on word-frequency distributions, i.e. the specific ‘Zipf’s-view’ and the non-specific ‘Randomness-view’, as is discussed. It is also noticed that the text-length transformation of a random book does have an exact scaling property precisely for the power-law index γ=1, as opposed to the Zipf’s exponent γ=2 and the implication of this exact scaling property is discussed. However a real text has γ>1 and as a consequence γ increases when shortening a real text. The connections to the predictions from the RGF (Random Group Formation) and to the infinite length-limit of a meta-book are also discussed. The difference between ‘curve-fitting’ and ‘predicting’ word-frequency distributions is stressed. It is pointed out that the question of randomness versus specifics for the distribution of outcomes in case of sufficiently complex systems has a much wider relevance than just the word-frequency example analyzed in the present work.

Keywords: Word-frequency distributions; Zipf’s law; Random Group Formation; Maximum entropy (search for similar items in EconPapers)
Date: 2016
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0378437115009413
Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:phsmap:v:444:y:2016:i:c:p:828-837

DOI: 10.1016/j.physa.2015.10.082

Access Statistics for this article

Physica A: Statistical Mechanics and its Applications is currently edited by K. A. Dawson, J. O. Indekeu, H.E. Stanley and C. Tsallis

More articles in Physica A: Statistical Mechanics and its Applications from Elsevier
Bibliographic data for series maintained by Catherine Liu ().