Bandit algorithms for policy learning: methods, implementation, and welfare-performance

Toru Kitagawa () and Jeff Rowley
Additional contact information
Toru Kitagawa: Brown University
Jeff Rowley: University College London

The Japanese Economic Review, 2024, vol. 75, issue 3, No 4, 407-447

Abstract: Abstract Static supervised learning—in which experimental data serves as a training sample for the estimation of an optimal treatment assignment policy—is a commonly assumed framework of policy learning. An arguably more realistic but challenging scenario is a dynamic setting in which the planner performs experimentation and exploitation simultaneously with subjects that arrive sequentially. This paper studies bandit algorithms for learning an optimal individualised treatment assignment policy. Specifically, we study applicability of the EXP4.P (Exponential weighting for Exploration and Exploitation with Experts) algorithm developed by Beygelzimer et al. (Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, JMLR Workshop and Conference Proceedings, pp 19–26, 2011) to policy learning. Assuming that the class of policies has a finite Vapnik–Chervonenkis dimension and that the number of subjects to be allocated is known, we present a high probability welfare-regret bound of the algorithm. To implement the algorithm, we use an incremental enumeration algorithm for hyperplane arrangements. We perform extensive numerical analysis to assess the algorithm’s sensitivity to its tuning parameters and its welfare-regret performance. Further simulation exercises are calibrated to the National Job Training Partnership Act (JTPA) Study sample to determine how the algorithm performs when applied to economic data. Our findings highlight various computational challenges and suggest that the limited welfare gain from the algorithm is due to substantial heterogeneity in causal effects in the JTPA data.

Keywords: Empirical Welfare Maximisation; Hyperplane arrangements; Reinforcement learning; Treatment choice (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://link.springer.com/10.1007/s42973-024-00165-6 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:jecrev:v:75:y:2024:i:3:d:10.1007_s42973-024-00165-6

Ordering information: This journal article can be ordered from
https://www.springer.com/journal/42973

DOI: 10.1007/s42973-024-00165-6

Access Statistics for this article

The Japanese Economic Review is currently edited by Michihiro Kandori

More articles in The Japanese Economic Review from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().