A nonlinear relationship between prediction errors and learning rates in human reinforcement-learning

Boluwatife Ikwunne, Jolie Parham and Erdem Pulcu

PLOS Computational Biology, 2025, vol. 21, issue 9, 1-21

Abstract: Reinforcement-learning (RL) models have been pivotal to our understanding of how agents perform learning-based adaptions in dynamically changing environments. However, the exact nature of the relationship (e.g., linear, logarithmic etc.) between key components of RL models such as prediction errors (PEs; the difference between the agent’s expectation and the actual outcome) and learning rates (a coefficient used by agents to update their beliefs about the environment) has not been studied in detail. Here, across (i) simulations, (ii) reanalyses of readily available datasets and (iii) a novel experiment, we demonstrate that the relationship between PEs and learning rates is (i) nonlinear over the PE/ learning rates space, and (ii) it can be accounted for by an exponential-logarithmic function that can transform the magnitude of PEs instantaneously to learning rates in a novel RL model. In line with the temporal predictions of this model, we show that physiological correlates of learning rates accumulate while learners observe the outcome of their choices and update their beliefs about the environment.Author summary: All living agents constantly learn and adapt to changes in their environments, a process normally hidden from observation and often understood through computational models. A key part of this is how we react to “prediction errors” – the difference between what we expect and what actually happens. These differences influence our “learning rate,” which is how quickly we update our beliefs about the world, and not much scientific work has been done on the exact relationship between prediction errors and learning rates. Our work demonstrates that this relationship is not always simple, or linear. Instead, we suggest that it is non-linear and depends on different types of uncertainty in the environment. Furthermore, physiological activity measured by recording pupil size during learning suggest that correlations linked to learning rates build up as we observe the outcomes of our actions and adjust our beliefs, supporting our proposed model accounting for how our brains use unexpected events to refine learning.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1013445 (text/html)
https://journals.plos.org/ploscompbiol/article/fil ... 13445&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pcbi00:1013445

DOI: 10.1371/journal.pcbi.1013445

Access Statistics for this article

More articles in PLOS Computational Biology from Public Library of Science
Bibliographic data for series maintained by ploscompbiol ().