Optimal Exploration-Exploitation in a Multi-armed-Bandit Problem with Non-stationary Rewards

Omar Besbes, Yonatan Gur and Assaf Zeevi
Additional contact information
Omar Besbes: Columbia University
Yonatan Gur: Stanford University
Assaf Zeevi: Columbia University

Research Papers from Stanford University, Graduate School of Business

Abstract: In a multi-armed bandit (MAB) problem a gambler needs to choose at each round of play one of K arms, each characterized by an unknown reward distribution. Reward realizations are only observed when an arm is selected, and the gambler's objective is to maximize his cumulative expected earnings over some given horizon of play T. To do this, the gambler needs to acquire information about arms (exploration) while simultaneously optimizing immediate rewards (exploitation); the price paid due to this trade off is often referred to as the regret, and the main question is how small can this price be as a function of the horizon length T. This problem has been studied extensively when the reward distributions do not change over time; an assumption that supports a sharp characterization of the regret, yet is often violated in practical settings. In this paper, we focus on a MAB formulation which allows for a broad range of temporal uncertainties in the rewards, while still maintaining mathematical tractability. We fully characterize the (regret) complexity of this class of MAB problems by establishing a direct link between the extent of allowable reward "variation" and the minimal achievable regret. Our analysis draws some connections between two rather disparate strands of literature: the adversarial and the stochastic MAB frameworks.

Date: 2014-08
New Economics Papers: this item is included in nep-mfd
References: Add references at CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://www.gsb.stanford.edu/faculty-research/worki ... d-bandit-problem-non

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ecl:stabus:3147

Access Statistics for this paper

More papers in Research Papers from Stanford University, Graduate School of Business Contact information at EDIRC.
Bibliographic data for series maintained by ().