Towards Faithful and Robust LLM Specialists for Evidence-Based Question-Answering

Tobias Schimanski, Jingwei Ni, Mathias Kraus, Elliott Ash and Markus Leippold
Additional contact information
Tobias Schimanski: University of Zurich
Jingwei Ni: ETH Zurich
Mathias Kraus: University of Erlangen-Nuremberg-Friedrich Alexander Universität Erlangen Nürnberg
Markus Leippold: University of Zurich; Swiss Finance Institute

No 24-94, Swiss Finance Institute Research Paper Series from Swiss Finance Institute

Abstract: Advances towards more faithful and traceable answers of Large Language Models (LLMs) are crucial for various research and practical endeavors. One avenue in reaching this goal is basing the answers on reliable sources. However, this Evidence-Based QA has proven to work insufficiently with LLMs in terms of citing the correct sources (source quality) and truthfully representing the information within sources (answer attributability). In this work, we systematically investigate how to robustly fine-tune LLMs for better source quality and answer attributability. Specifically, we introduce a data generation pipeline with automated data quality filters, which can synthesize diversified high-quality training and testing data at scale. We further introduce four test sets to benchmark the robustness of fine-tuned specialist models. Extensive evaluation shows that fine-tuning on synthetic data improves performance on both in- and out-of-distribution. Furthermore, we show that data quality, which can be drastically improved by proposed quality filters, matters more than quantity in improving Evidence-Based QA.

Pages: 20 pages
Date: 2024-03
New Economics Papers: this item is included in nep-big and nep-cmp
References: Add references at CitEc
Citations:

Downloads: (external link)
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4728973 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:chf:rpseri:rp2494

Access Statistics for this paper

More papers in Swiss Finance Institute Research Paper Series from Swiss Finance Institute Contact information at EDIRC.
Bibliographic data for series maintained by Ridima Mittal ().