FAIR: Fairness‐aware information retrieval evaluation

Ruoyuan Gao, Yingqiang Ge and Chirag Shah

Journal of the Association for Information Science & Technology, 2022, vol. 73, issue 10, 1461-1473

Abstract: With the emerging needs of creating fairness‐aware solutions for search and recommendation systems, a daunting challenge exists of evaluating such solutions. While many of the traditional information retrieval (IR) metrics can capture the relevance, diversity, and novelty for the utility with respect to users, they are not suitable for inferring whether the presented results are fair from the perspective of responsible information exposure. On the other hand, existing fairness metrics do not account for user utility or do not measure it adequately. To address this problem, we propose a new metric called FAIR. By unifying standard IR metrics and fairness measures into an integrated metric, this metric offers a new perspective for evaluating fairness‐aware ranking results. Based on this metric, we developed an effective ranking algorithm that jointly optimized user utility and fairness. The experimental results showed that our FAIR metric could highlight results with good user utility and fair information exposure. We showed how FAIR related to a set of existing utility and fairness metrics and demonstrated the effectiveness of our FAIR‐based algorithm. We believe our work opens up a new direction of pursuing a metric for evaluating and implementing the FAIR systems.

Date: 2022
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1002/asi.24648

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:jinfst:v:73:y:2022:i:10:p:1461-1473

Ordering information: This journal article can be ordered from
http://www.blackwell ... bs.asp?ref=2330-1635

Access Statistics for this article

More articles in Journal of the Association for Information Science & Technology from Association for Information Science & Technology
Bibliographic data for series maintained by Wiley Content Delivery ().