Do Employers “Walk the Talk†After All? An Illustration of Methods for Assessing Signals in Underpowered Designs

Jonathan R. Brauer, Jacob C. Day and Brittany M. Hammond

Sociological Methods & Research, 2021, vol. 50, issue 4, 1801-1841

Abstract: This article presents two alternative methods to null hypothesis significance testing (NHST) for improving inferences from underpowered research designs. Post hoc design analysis (PHDA) assesses whether an NHST analysis generating null findings might otherwise have had sufficient power to detect effects of plausible magnitudes. Bayesian analysis with default priors offers advantages over NHST for assessing null findings and detecting signals in underpowered data. Both methods are illustrated by application to Pager and Quillian’s influential study on attitude-behavior correspondence. PHDA results suggest the original study lacked sufficient power to detect strong associations between employers’ attitudes and behaviors. Bayesian analysis confirms strong attitude–behavior associations cannot be ruled out given the data. Together, these results question a frequently cited conclusion about attitude–behavior incongruence in survey vignettes. Overall, the examples illustrate how these analytical tools can be useful for describing uncertainty surrounding estimates and for improving substantive and theoretical debates across sociology.

Keywords: post hoc design analysis; Bayesian analysis; attitude–behavior correspondence; null findings; underpowered designs; survey vignettes; audit studies; statistical power (search for similar items in EconPapers)
Date: 2021
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/0049124119826158 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:somere:v:50:y:2021:i:4:p:1801-1841

DOI: 10.1177/0049124119826158

Access Statistics for this article

More articles in Sociological Methods & Research
Bibliographic data for series maintained by SAGE Publications ().