Policy Evaluation Beyond Significance Testing: Evidence from Chicago's ShotSpotter Contract

Terence Chau, Philip J Cook, Javier Lopez, Jens Ludwig, Alejandro Roemer and Adam K. Soliman

No 36005, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Program impact estimates that are not statistically significant are often treated as providing little guidance for policy choice, even when economically important effects remain plausible. In such cases, the decision-relevant task is not simply to test whether the effect differs from zero, but to estimate the probability distribution of programmatic effects. The importance of going beyond hypothesis testing is especially clear when evaluations are underpowered relative to the effect needed for benefits to exceed costs. We illustrate this point using Chicago’s contract for ShotSpotter, an acoustical gunshot detection system that reduces police response times and may therefore have improved survival among shooting victims. Using a geographic RD design, we estimate that ShotSpotter reduced the shooting fatality rate by 2.5 percentage points, equivalent to roughly 49 fewer deaths per year. Although the estimate is not statistically significant, its implied benefit far exceeds the cost of the service. To have enough statistical power under the hypothesis testing approach to detect an effect of this size would require 87 years’ worth of Chicago shooting data. We instead implement a decision-relevant perspective using Bayesian analysis, treating the life-saving effect as a random variable. We find positive expected net benefits, with posterior probabilities that benefits exceed costs ranging from 70% under a flat prior and a $0.5 million value of statistical life to 99.6% under a no-harm prior and a $13.6 million value. The results illustrate how reliance on statistical significance can discard decision-relevant evidence when economically meaningful effects are difficult to detect precisely.

JEL-codes: C10 C12 C44 K42 (search for similar items in EconPapers)
Date: 2026-11
Note: CH DEV ED EH LE LS PE
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.nber.org/papers/w36005.pdf (application/pdf)
Access to the full text is generally limited to series subscribers, however if the top level domain of the client browser is in a developing country or transition economy free access is provided. More information about subscriptions and free access is available at http://www.nber.org/wwphelp.html. Free access is also available to older working papers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:36005

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w36005
The price is Paper copy available by mail.

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().