When Samples Shape Significance

Sonia Ale, Md Shafiqul Islam, Lester Lusher (), Huda Osman and Jacob Stenstrom
Additional contact information
Sonia Ale: University of Pittsburgh
Md Shafiqul Islam: University of Pittsburgh
Lester Lusher: University of Pittsburgh
Huda Osman: University of Pittsburgh
Jacob Stenstrom: University of Pittsburgh

No 18892, IZA Discussion Papers from IZA Network @ LISER

Abstract: Random sampling yields unbiased estimates, yet sampling variation alone can produce incorrect conclusions. Using a setting where repeated draws from the same data-generating process are observable, we replicate findings from 24 papers in top economics journals and re-estimate each result using 40 independent resamples. Across 3,640 resampled results, t-statistics shrink by 31%, frequently leading to lost significance. Proxies for researcher degrees of freedom and sampling noise predict reduced significance. Our exercise provides novel evidence quantifying the effect of sampling variability on empirical research and highlights how statistical evidence can vary even when holding research design fixed.

Keywords: sampling variation; statistical significance; reproducibility; publication bias; Google Trends (search for similar items in EconPapers)
JEL-codes: C12 C18 C80 (search for similar items in EconPapers)
Date: 2026-08
References: Add references at CitEc
Citations:

Downloads: (external link)
https://docs.iza.org/dp18892.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:iza:izadps:dp18892

Access Statistics for this paper

More papers in IZA Discussion Papers from IZA Network @ LISER Contact information at EDIRC.
Bibliographic data for series maintained by Mark Fallak ().