Human uncertainty in explicit user feedback and its impact on the comparative evaluations of accurate prediction and personalisation

Kevin Jasberg and Sergej Sizov

Behaviour and Information Technology, 2020, vol. 39, issue 5, 544-577

Abstract: In this article, we report on the lack of reliability of explicit user feedback and its interpretation in the light of system evaluation. It is known, that given feedback strongly depends on the situational context. But also when many contextual factors are held constant, user feedback still proves to be unreliable. This impacts the evaluation of predictive algorithms since it is not clear whether a deviation between a user response and its corresponding prediction can be seen as a flaw by the system or just as usual ‘human uncertainty’. As a result, the perspective on the evaluation of adaptive systems basically changes. The main goal of this article is to demonstrate that simply increasing the amount of explicit feedback is not the key to sustainable system design innovation, as long as that information is not appropriately evaluated. To this end, we will exploit a novel probabilistic approach of processing user feedback and identify biasing effects on accuracy metrics, error probabilities for system rankings as well as natural limitations of evaluation. Finally, we will discuss possible solution strategies and give advice for handling explicit user feedback that is associated with uncertainty.

Date: 2020
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/0144929X.2019.1604804 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:tbitxx:v:39:y:2020:i:5:p:544-577

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/tbit20

DOI: 10.1080/0144929X.2019.1604804

Access Statistics for this article

Behaviour and Information Technology is currently edited by Dr Panos P Markopoulos

More articles in Behaviour and Information Technology from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().