Evaluating the generalizability of the COVID States survey — a large-scale, non-probability survey

Jason Radford, Jon Green, Alexi Quintana, Alauna Safarpour, Matthew D Simonson, Matthew Baum, David Lazer, Katherine Ognyanova, James Druckman and Roy Perlis
Additional contact information
Alauna Safarpour: Harvard University
Katherine Ognyanova: Rutgers University

No cwkg7, OSF Preprints from Center for Open Science

Abstract: COVID-19 fundamentally changed the world in a matter of months. To understand how it was impacting life in the United States, we fielded a non-probability survey in all 50 states concerning people's attitudes, beliefs, and behaviors, designed to be representative at the state level. Here, we evaluate the generalizability of this study by assessing the representativeness and convergent validity of our estimates. First, we evaluate the representativeness of the sample by comparing it to baseline estimates and auditing the size of the weights we use to reduce bias. We find our sample is diverse and most weights are below levels of concern with the exception of Hispanic respondents. Second, we assess the convergent validity of our survey by evaluating how our estimates of attitudes, behaviors, and opinions compare to estimates from other surveys and administrative data. Third, we perform a direct comparison of our results to the Kaiser Family Foundation’s probability-based COVID-19 Vaccine Monitor. Overall, our estimates deviate from others by 1%-7% with the larger differences stemming from states with small populations and few other data sources and estimates from items with differing question wording or response choices. Here, we put forward a standard for evaluating the representativeness of surveys, non-probability or otherwise.

Date: 2022-03-07
New Economics Papers: this item is included in nep-hea
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/62264008558e6004d32a0a6a/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:cwkg7

DOI: 10.31219/osf.io/cwkg7

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().