Data Quality in Estimates from Probability-Based Online Panels: Systematic Review and Meta-Analysis

Andrea Ivanovska, Michael Bosnjak and Vasja Vehovar

Acta Informatica Pragensia, vol. preprint

Abstract: Background: General population surveys now increasingly use nonprobability samples from access panels instead of probability-based methods, which often leads to lower-quality estimates. In response, many official and academic surveys have adopted probability-based online panels (PBOPs), which use probability sampling and retain participants for follow-up surveys. While these panels reduce costs compared to one-time surveys, they still face low response rates and other challenges that may affect data quality.Objective: This study aimed to assess the accuracy of PBOPs by synthesising evidence on relative bias (RB), and to examine how RB varies by country, domain, measurement level, and item sensitivity.Methods: A systematic review yielded 44 eligible studies from 12 countries, and 1,897 effect sizes of absolute RB from studies that compared PBOP estimates to benchmarks. A three-level random effects meta-analytic model accounted for variance across studies, within studies and sampling variance. Moderator analyses evaluated the influence of country, item topic, measurement level and sensitivity on RB. Sensitivity analyses excluded the top 5% of RB outliers to test robustness.Results: The pooled RB was 23.14% (95% CI: 18.38%-27.91%) and heterogeneous. Most variance was attributed to within-study item-level differences. Country and topic did not significantly moderate RB. Items with high topic sensitivity had significantly higher RB (+19.33%) than items with no sensitivity. Ordinal items had significantly lower RB than nominal (-14.90%). However, when sensitivity and measurement level were modelled together, substantial residual heterogeneity remained.Conclusion: While PBOPs offer cost and logistical advantages, they require careful design considerations to lower substantial bias, especially regarding item sensitivity and measurement scale. PBOPs may not be suitable for certain question types, like sensitive or low-prevalence behaviours, especially when high accuracy is needed. Improved methodological planning and innovations are needed to improve PBOP data quality.

Keywords: Online surveys; Probability-based online panels; Data quality; Relative bias; Meta-analysis (search for similar items in EconPapers)
References: Add references at CitEc
Citations:

Downloads: (external link)
http://aip.vse.cz/doi/10.18267/j.aip.279.html (text/html)
free of charge

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:prg:jnlaip:v:preprint:id:279

Ordering information: This journal article can be ordered from
Redakce Acta Informatica Pragensia, Katedra systémové analýzy, Vysoká škola ekonomická v Praze, nám. W. Churchilla 4, 130 67 Praha 3
http://aip.vse.cz

DOI: 10.18267/j.aip.279

Access Statistics for this article

Acta Informatica Pragensia is currently edited by Editorial Office

More articles in Acta Informatica Pragensia from Prague University of Economics and Business Contact information at EDIRC.
Bibliographic data for series maintained by Stanislav Vojir ().