Expectation-maximization for Bayes-adaptive POMDPs

Erik P Vargo and Randy Cogill
Additional contact information
Erik P Vargo: University of Virginia, Charlottesville, USA
Randy Cogill: IBM Research – Ireland, Dublin, Ireland

Journal of the Operational Research Society, 2015, vol. 66, issue 10, 1605-1623

Abstract: Bayes-adaptive POMDPs (BAPOMDPs) are partially observable Markov decision problems in which uncertainty in the state-transition and observation-emission probabilities can be captured by a prior distribution over the model parameters. Existing approaches to solving BAPOMDPs rely on model and trajectory sampling to guide exploration and, because of the curse of dimensionality, do not scale well when the degree of model uncertainty is large. In this paper, we begin by presenting two expectation-maximization (EM) approaches to solving BAPOMPs via finite-state controller (FSC) optimization, which at their foundation are extensions of existing EM algorithms for BAMDPs to the more general BAPOMDP setting. The first is a sampling-based EM algorithm that optimizes over a finite number of models drawn from the BAPOMDP prior, and as such is only appropriate for smaller problems with limited model uncertainty; the second approach leverages variational Bayesian methods to ensure tractability without sampling, and is most appropriate for larger domains with greater model uncertainty. Our primary novel contribution is the derivation of the constrained VB-EM algorithm, which addresses an unfavourable preference that often arises towards a certain class of policies when applying the standard VB-EM algorithm. Through an empirical study we show that the sampling-based EM algorithm is competitive with more conventional sampling-based approaches in smaller domains, and that our novel constrained VB-EM algorithm can generate quality solutions in larger domains where sampling-based approaches are no longer viable.

Date: 2015
References: Add references at CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://www.palgrave-journals.com/jors/journal/v66/n10/pdf/jors201549a.pdf Link to full text PDF (application/pdf)
http://www.palgrave-journals.com/jors/journal/v66/n10/full/jors201549a.html Link to full text HTML (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:pal:jorsoc:v:66:y:2015:i:10:p:1605-1623

Ordering information: This journal article can be ordered from
http://www.springer. ... search/journal/41274

Access Statistics for this article

Journal of the Operational Research Society is currently edited by Tom Archibald and Jonathan Crook

More articles in Journal of the Operational Research Society from Palgrave Macmillan, The OR Society
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().