A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator

Magdalena Smyk, Joanna Tyrowicz and Lucas van der Velde

No 11503, IZA Discussion Papers from Institute of Labor Economics (IZA)

Abstract: We investigate the reliability of data from the Wage Indicator (WI), the largest online survey on earnings and working conditions. Comparing WI to nationally representative data sources for 17 countries reveals that participants of WI are not likely to have been representatively drawn from the respective populations. Previous literature has proposed to utilize weights based on inverse propensity scores, but this procedure was shown to leave reweighted WI samples different from the benchmark nationally representative data. We propose a novel procedure, building on covariate balancing propensity score, which achieves complete reweighting of the WI data, making it able to replicate the structure of nationally representative samples on observable characteristics. While rebalancing assures the match between WI and representative benchmark data sources, we show that the wage schedules remain different for a large group of countries. Using the example of a Mincerian wage regression, we find that in more than a third of the cases, our proposed novel reweighting assures that estimates obtained on WI data are not biased relative to nationally representative data. However, in the remaining 60% of the analyzed 95 datasets systematic differences in the estimated coefficients of the Mincerian wage regression between WI and nationally representative data persists even after reweighting. We provide some intuition about the reasons behind these biases. Notably, objective factors such as access to the Internet or richness appear to matter, but self-selection (on unobservable characteristics) among WI participants appears to constitute an important source of bias.

Keywords: propensity score matching; online surveys; Wage Indicator; weights (search for similar items in EconPapers)
JEL-codes: C81 J30 J31 (search for similar items in EconPapers)
Pages: 39 pages
Date: 2018-04
New Economics Papers: this item is included in nep-lma
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (7)

Published - published in: Sociological Methods and Research, 2021, 50 (1), 429-464

Downloads: (external link)
https://docs.iza.org/dp11503.pdf (application/pdf)

Related works:
Journal Article: A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator (2021)
Working Paper: A cautionary note on the reliability of the online survey data - the case of Wage Indicator (2018)
Working Paper: A cautionary note on the reliability of the online survey data – the case of Wage Indicator (2018)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:iza:izadps:dp11503

Ordering information: This working paper can be ordered from
IZA, Margard Ody, P.O. Box 7240, D-53072 Bonn, Germany

Access Statistics for this paper

More papers in IZA Discussion Papers from Institute of Labor Economics (IZA) IZA, P.O. Box 7240, D-53072 Bonn, Germany. Contact information at EDIRC.
Bibliographic data for series maintained by Holger Hinte ().