Estimating the Impact of Emergency Assistance on Educational Progress for Low-Income Adults: Experimental and Nonexperimental Evidence

Daniel Litwok

Evaluation Review, 2023, vol. 47, issue 2, 231-263

Abstract: Methods for estimating causal impact aim to either remove or reduce bias. This study estimates the degree of bias reduction obtained from regression adjustment and propensity score methods when only a weak set of predictors are available. The study uses an experimental test of providing emergency financial assistance to participants in a job training program to estimate an experimental benchmark and compares it to nonexperimental estimates of the impact of receiving assistance. When estimating the impact of receiving assistance, those who received it constitute the treatment group. The study explores two different comparison groups: those who could have (because they were assigned to the experimental treatment group) but did not receive emergency assistance; and those who could not receive emergency assistance because they were randomly assigned to the experimental control group. It uses these groups to estimate impacts by applying three estimation strategies: unadjusted mean comparison, regression adjustment, and inverse propensity weighting. It then compares these estimates to the experimental benchmark using statistical tests recommended by the within-study comparison literature. The nonexperimental approaches to addressing selection bias suggest large positive impacts. These are statistically different from the experimental benchmark, which shows that receipt of emergency assistance does not improve educational progress. Further, over 90% of the bias from a simple comparison of means remains. Unless a stronger set of predictors are available, future evaluations of such interventions should be wary of relying on these methods for either unbiased estimation of impacts or bias reduction.

Keywords: nonexperimental evaluation; experimental evaluation; treatment on the treated; within-study comparison; emergency assistance (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/0193841X221118454 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:evarev:v:47:y:2023:i:2:p:231-263

DOI: 10.1177/0193841X221118454

Access Statistics for this article

More articles in Evaluation Review
Bibliographic data for series maintained by SAGE Publications ().