Downloading Wisdom from Online Crowds

Albert Saiz and Uri Simonsohn ()
Additional contact information
Uri Simonsohn: University of California, San Diego

No 3809, IZA Discussion Papers from Institute of Labor Economics (IZA)

Abstract: The internet and other large textual databases contain billions of documents: is there useful information in the number of documents written about different topics? We propose, based on the premise that the occurrence of a phenomenon increases the likelihood that people write about it, that the relative frequency of documents discussing a phenomenon can be used to proxy for the corresponding occurrence-frequency. After establishing the conditions under which such proxying is likely to be successful, we construct proxies for a number of demographic variables in the US and for corruption across countries and US states and cities, obtaining average correlations with occurrence-frequencies of 0.47 and 0.61 respectively. We also replicate results from two separate published papers establishing the correlates of corruption at both the state and country level. Finally, we construct the first index of corruption in US cities and study its correlates.

Keywords: proxy variables; document-frequency; textual databases; internet (search for similar items in EconPapers)
JEL-codes: B40 C81 J11 (search for similar items in EconPapers)
Pages: 54 pages
Date: 2008-10
New Economics Papers: this item is included in nep-ict
References: Add references at CitEc
Citations: View citations in EconPapers (10)

Published - revised version published as 'Proxying for Unobservable Variables with Internet Document-Frequency' in: Journal of the European Economic Association, 2013, 11 (1), 137–165

Downloads: (external link)
https://docs.iza.org/dp3809.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:iza:izadps:dp3809

Ordering information: This working paper can be ordered from
IZA, Margard Ody, P.O. Box 7240, D-53072 Bonn, Germany

Access Statistics for this paper

More papers in IZA Discussion Papers from Institute of Labor Economics (IZA) IZA, P.O. Box 7240, D-53072 Bonn, Germany. Contact information at EDIRC.
Bibliographic data for series maintained by Holger Hinte ().