Introduction to EBPM (Evidence-Based Policy Making) Episode 4: Design Matters More Than Analysis (Supplementary Notes on Impact Evaluation) (Japanese)

Yoichi Sekizawa

Policy Discussion Papers (Japanese) from Research Institute of Economy, Trade and Industry (RIETI)

Abstract: In understanding impact evaluation, several important cautions should be noted: (1) statistical significance, a concept frequently used in evaluation, should not be overemphasized; (2) results that once showed a significant effect may fail to replicate; and (3) the presence or magnitude of effects may differ across units (e.g., individuals, firms). Although methods such as instrumental variable estimations (IV) and propensity score matching (PSM) are frequently used for impact evaluation, there are cases where it is difficult to determine whether the results are reliable. Practitioners in policy settings are therefore advised to approach these methods with appropriate caution. To conduct effective impact evaluation, preparation at the policy design stage is crucial—indeed, design matters more than analysis. It is desirable for those involved in policy design to consider appropriate strategies such as creating groups that receive interventions earlier than others based on random choice, and avoiding institutional designs that simply include all applicants as beneficiaries of the programs.

Pages: 13 pages
Date: 2026-05
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.rieti.go.jp/jp/publications/pdp/26p011.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eti:rpdpjp:26011

Access Statistics for this paper

More papers in Policy Discussion Papers (Japanese) from Research Institute of Economy, Trade and Industry (RIETI) Contact information at EDIRC.
Bibliographic data for series maintained by TANIMOTO, Toko ().