EconPapers    
Economics at your fingertips  
 

Identifiability and Generalizability from Multiple Experts in Inverse Reinforcement Learning

Paul Rolland, Luca Viano, Norman Schürhoff, Boris Nikolov and Volkan Cevher
Additional contact information
Paul Rolland: Ecole Polytechnique Fédérale de Lausanne
Luca Viano: Ecole Polytechnique Fédérale de Lausanne
Norman Schürhoff: Swiss Finance Institute - HEC Lausanne
Boris Nikolov: University of Lausanne; Swiss Finance Institute; European Corporate Governance Institute (ECGI)
Volkan Cevher: Ecole Polytechnique Fédérale de Lausanne

Authors registered in the RePEc Author Service: Norman Schuerhoff

No 22-79, Swiss Finance Institute Research Paper Series from Swiss Finance Institute

Abstract: While Reinforcement Learning (RL) aims to train an agent from a reward function in a given environment, Inverse Reinforcement Learning (IRL) seeks to recover the reward function from observing an expert’s behavior. It is well known that, in general, various reward functions can lead to the same optimal policy, and hence, IRL is ill-defined. However, [1] showed that, if we observe two or more experts with different discount factors or acting in different environments, the reward function can under certain conditions be identified up to a constant. This work starts by showing an equivalent identifiability statement from multiple experts in tabular MDPs based on a rank condition, which is easily verifiable and is shown to be also necessary. We then extend our result to various different scenarios, i.e., we characterize reward identifiability in the case where the reward function can be represented as a linear combination of given features, making it more interpretable, or when we have access to approximate transition matrices. Even when the reward is not identifiable, we provide conditions characterizing when data on multiple experts in a given environment allows to generalize and train an optimal agent in a new environment. Our theoretical results on reward identifiability and generalizability are validated in various numerical experiments.

Pages: 28 pages
Date: 2022-10
New Economics Papers: this item is included in nep-cmp
References: Add references at CitEc
Citations:

Downloads: (external link)
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4251437 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:chf:rpseri:rp2279

Access Statistics for this paper

More papers in Swiss Finance Institute Research Paper Series from Swiss Finance Institute Contact information at EDIRC.
Bibliographic data for series maintained by Ridima Mittal ().

 
Page updated 2025-03-22
Handle: RePEc:chf:rpseri:rp2279