Environmental uncertainty and the advantage of impulsive choice strategies

Diana C Burk and Bruno B Averbeck

PLOS Computational Biology, 2023, vol. 19, issue 1, 1-34

Abstract: Choice impulsivity is characterized by the choice of immediate, smaller reward options over future, larger reward options, and is often thought to be associated with negative life outcomes. However, some environments make future rewards more uncertain, and in these environments impulsive choices can be beneficial. Here we examined the conditions under which impulsive vs. non-impulsive decision strategies would be advantageous. We used Markov Decision Processes (MDPs) to model three common decision-making tasks: Temporal Discounting, Information Sampling, and an Explore-Exploit task. We manipulated environmental variables to create circumstances where future outcomes were relatively uncertain. We then manipulated the discount factor of an MDP agent, which affects the value of immediate versus future rewards, to model impulsive and non-impulsive behavior. This allowed us to examine the performance of impulsive and non-impulsive agents in more or less predictable environments. In Temporal Discounting, we manipulated the transition probability to delayed rewards and found that the agent with the lower discount factor (i.e. the impulsive agent) collected more average reward than the agent with a higher discount factor (the non-impulsive agent) by selecting immediate reward options when the probability of receiving the future reward was low. In the Information Sampling task, we manipulated the amount of information obtained with each sample. When sampling led to small information gains, the impulsive MDP agent collected more average reward than the non-impulsive agent. Third, in the Explore-Exploit task, we manipulated the substitution rate for novel options. When the substitution rate was high, the impulsive agent again performed better than the non-impulsive agent, as it explored the novel options less and instead exploited options with known reward values. The results of these analyses show that impulsivity can be advantageous in environments that are unexpectedly uncertain.Author summary: Impulsive choice behavior, or valuing immediate smaller rewards over larger, delayed rewards, is typically considered to be detrimental in decision-making. In this study, we use Markov Decision Processes (MDPs) to demonstrate that impulsive choices can be beneficial in three common decision-making tasks: Temporal Discounting, Information Sampling and an Explore-Exploit task. Specifically, we found that when a task environment is more uncertain than expected, an impulsive agent can collect more average reward than a non-impulsive agent. Our work suggests that impulsivity is not inherently negative. Valuing immediate rewards over delayed rewards can be an adaptive strategy when faced with uncertainty.

Date: 2023
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010873 (text/html)
https://journals.plos.org/ploscompbiol/article/fil ... 10873&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pcbi00:1010873

DOI: 10.1371/journal.pcbi.1010873

Access Statistics for this article

More articles in PLOS Computational Biology from Public Library of Science
Bibliographic data for series maintained by ploscompbiol ().