To be or not to be in the sample? On using manipulation checks in experimental accounting research

Peter Kotzian, Thomas Stoeber, Florian Hoos and Barbara E. Weissenberger

Accounting Research Journal, 2020, vol. 33, issue 3, 469-482

Abstract: Purpose - Manipulation checks are a recommended for experimental accounting research. Usage of information gained by manipulation checks varies. In some studies, participants who failed the manipulation check are removed from the sample. Other studies report the results of the manipulation checks but still use the full sample. Some authors recommend removing participants who failed the manipulation check as a means to increase the power of the statistical analysis. Others warn that removing these participants endangers the randomization as a crucial precondition for gaining valid insights from experimental research. Until now, there is little research on how sensitive results react to exclusion of participants. The purpose of this paper is to study the effects of removing participants failing the manipulation checks on the evaluation of a hypothesis and the development of alternative usages of the information gained from manipulation checks. Design/methodology/approach - Based on an analytical model and a simulation, the authors show how removing participants who fail the manipulation check affects experimental findings. Findings - Simulations show that statistical results and conclusions drawn from an experiment differ substantially, depending on whether participants who failed the manipulation check are removed from the sample. As the participants who are removed are no random sub-sample, but share a certain property, the experimental results react strongly, typically showing significant results, where there are actually none. Originality/value - This paper is, to the best of the authors’ knowledge, the first to address the sensitivity of experimental results to removing participants who fail the manipulation check from the sample and the implications for the validity of conclusions drawn from experimental accounting research. This paper’s contribution is a better way of using information gained in the manipulation check in the statistical analysis of the experimental data.

Keywords: Experimental accounting research; Manipulation checks; Replicability (search for similar items in EconPapers)
Date: 2020
References: Add references at CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
https://www.emerald.com/insight/content/doi/10.110 ... d&utm_campaign=repec (text/html)
https://www.emerald.com/insight/content/doi/10.110 ... d&utm_campaign=repec (application/pdf)
Access to full text is restricted to subscribers

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eme:arjpps:arj-06-2019-0128

DOI: 10.1108/ARJ-06-2019-0128

Access Statistics for this article

Accounting Research Journal is currently edited by Professor Reza Monem

More articles in Accounting Research Journal from Emerald Group Publishing Limited
Bibliographic data for series maintained by Emerald Support ().