Estimation of country-level effects in cross-national survey research using multilevel modelling: The role of statistical power

Elena Damian, Bart Meuleman and Wim van Oorschot
Additional contact information
Elena Damian: Sciensano

No m94kh, OSF Preprints from Center for Open Science

Abstract: Multilevel regression analysis is one of the most popular types of analyses in cross-national social studies. However, since its early applications, there have been constant concerns about the relatively small numbers of countries in cross-national surveys and its ability to produce unbiased and accurate country-level effects. A recent review of Bryan and Jenkins (2016) highlights that there are still no clear rules of thumb regarding the minimum number of countries needed. The current recommendations vary from 15 to 50 countries, depending on model complexity. This paper aims to offer a better understanding regarding the consequences of group-level sample size, model complexity, effect size, and estimator procedure on the precision to estimate country-level effects in cross-national studies. The accuracy criteria considered are statistical power, relative parameter bias, relative standard error bias, and convergence rates. We pay special attention to statistical power - a key criteria that has been largely neglected in past research. The results of our Monte Carlo simulation study indicate that the small number of countries found in cross-national surveys seriously affects the accuracy of group-level estimates. Specifically, while a sample size of 30 countries is sufficient to detect large population effects (.5), the probability of detecting a medium (.25) or a small effect (.10) is .4 or .2, respectively. The number of additional group-level variables (i.e., model complexity) included in the model does not disturb the relationship between sample size and statistical power. Hence, adding contextual variables one by one does not increase the power to estimate a certain effect if the sample size is small. Even though we find that Bayesian models have more accurate estimates, there are no notable differences in statistical power between Maximum Likelihood and Bayesian models.

Date: 2022-08-19
New Economics Papers: this item is included in nep-ecm
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/62ffd09980b0d80412fe92da/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:m94kh

DOI: 10.31219/osf.io/m94kh

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().