A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator

Magdalena Smyk, Joanna Tyrowicz and Lucas van der Velde

Sociological Methods & Research, 2021, vol. 50, issue 1, 429-464

Abstract: We investigate the reliability of data from the Wage Indicator (WI), the largest online survey on earnings and working conditions. Comparing WI to nationally representative data sources for 17 countries reveals that participants of WI are not likely to have been representatively drawn from the respective populations. Previous literature has proposed to utilize weights based on inverse propensity scores, but this procedure was shown to leave reweighted WI samples different from the benchmark nationally representative data. We propose a novel procedure, building on covariate balancing propensity score, which achieves complete reweighting of the WI data, making it able to replicate the structure of nationally representative samples on observable characteristics. While rebalancing assures the match between WI and representative benchmark data sources, we show that the wage schedules remain different for a large group of countries. Using the example of a Mincerian wage regression, we find that in more than a third of the cases, our proposed novel reweighting assures that estimates obtained on WI data are not biased relative to nationally representative data. However, in the remaining 60 percent of the analyzed 95 data sets, systematic differences in the estimated coefficients of the Mincerian wage regression between WI and nationally representative data persist even after reweighting. We provide some intuition about the reasons behind these biases. Notably, objective factors such as access to the Internet or richness appear to matter, but self-selection (on unobservable characteristics) among WI participants appears to constitute an important source of bias.

Keywords: wage indicator; online surveys; propensity score matching; weights (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/0049124118782538 (text/html)

Related works:
Working Paper: A cautionary note on the reliability of the online survey data - the case of Wage Indicator (2018)
Working Paper: A cautionary note on the reliability of the online survey data – the case of Wage Indicator (2018)
Working Paper: A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator (2018)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:somere:v:50:y:2021:i:1:p:429-464

DOI: 10.1177/0049124118782538

Access Statistics for this article

More articles in Sociological Methods & Research
Bibliographic data for series maintained by SAGE Publications ().