Intrinsic rewards explain context-sensitive valuation in reinforcement learning

Gaia Molinaro and Anne G E Collins

PLOS Biology, 2023, vol. 21, issue 7, 1-31

Abstract: When observing the outcome of a choice, people are sensitive to the choice’s context, such that the experienced value of an option depends on the alternatives: getting $1 when the possibilities were 0 or 1 feels much better than when the possibilities were 1 or 10. Context-sensitive valuation has been documented within reinforcement learning (RL) tasks, in which values are learned from experience through trial and error. Range adaptation, wherein options are rescaled according to the range of values yielded by available options, has been proposed to account for this phenomenon. However, we propose that other mechanisms—reflecting a different theoretical viewpoint—may also explain this phenomenon. Specifically, we theorize that internally defined goals play a crucial role in shaping the subjective value attributed to any given option. Motivated by this theory, we develop a new “intrinsically enhanced” RL model, which combines extrinsically provided rewards with internally generated signals of goal achievement as a teaching signal. Across 7 different studies (including previously published data sets as well as a novel, preregistered experiment with replication and control studies), we show that the intrinsically enhanced model can explain context-sensitive valuation as well as, or better than, range adaptation. Our findings indicate a more prominent role of intrinsic, goal-dependent rewards than previously recognized within formal models of human RL. By integrating internally generated signals of reward, standard RL theories should better account for human behavior, including context-sensitive valuation and beyond.When observing the outcome of a choice, people are sensitive to the choice’s context, such that the experienced value of an option depends on the alternatives. Computational analysis of seven different studies suggests that internally defined goals play a crucial role in shaping the subjective value attributed to available options in reinforcement learning.

Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002201 (text/html)
https://journals.plos.org/plosbiology/article/file ... 02201&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pbio00:3002201

DOI: 10.1371/journal.pbio.3002201

Access Statistics for this article

More articles in PLOS Biology from Public Library of Science
Bibliographic data for series maintained by plosbiology ().