LLM, Trust and Performance: case study of a Competitive Intelligence solution

LLM, confiance et performance: étude de cas d’une solution d’intelligence économique

Anna Nesvijevskaia () and Stefan Berechet
Additional contact information
Anna Nesvijevskaia: DICEN-IDF - Dispositifs d'Information et de Communication à l'Ère du Numérique - Paris Île-de-France - UPN - Université Paris Nanterre - Cnam - Conservatoire National des Arts et Métiers [Cnam] - Université Gustave Eiffel, HEG - Haute Ecole de Gestion de Genève, ISI 4C - Intelligence Swiss Initiative - HEG - Haute Ecole de Gestion de Genève
Stefan Berechet: Quinten

Post-Print from HAL

Abstract: The recent democratization of Large Language Models (LLMs) has profoundly transformed access to and exploitation of external data, particularly in the field of Strategic and Competitive Intelligence (CI). While LLM-based solutions offer new opportunities to enhance decision-making, innovation detection, and information processing, they also raise significant challenges regarding their evaluation, notably in terms of performance, trust, and operational impact. Existing evaluation frameworks, largely inherited from traditional machine learning, appear insufficient to capture the multifaceted, subjective, and evolving nature of LLM-driven uses. This paper explores the evaluation of LLM-based CI solutions through an in-depth case study of a French data science project aimed at supporting innovation scouting for enterprises. Building on a review of state-of-the-art LLM evaluation metrics and methodologies, the study adopts an anthropocentric qualitative approach, based on eleven semi-structured interviews conducted with data and business stakeholders involved in the co-design of the solution. The analysis follows the CRISP-DM framework to examine how evaluation activities are embedded throughout the project lifecycle. The findings highlight an extreme complexity in defining and operationalizing evaluation criteria, driven by the coexistence of multiple objectives such as relevance, exhaustiveness, creativity, usability, and strategic impact. Beyond statistical metrics, the study reveals the growing importance of behavioral and psychological dimensions, particularly user trust, which strongly influence adoption and perceived value but remain costly and difficult to measure over time. Moreover, the rapid evolution of LLM technologies and the constraints related to sovereignty, confidentiality, and intellectual property further complicate model selection, benchmarking, and long-term governance. The paper concludes by discussing emerging challenges for performance assessment, project arbitration, and the balance between standardization and flexibility of practices. It calls for renewed evaluation frameworks that integrate technical, organizational, and human factors, and outlines avenues for future research on sustainable and interpretable evaluation of LLM-based CI systems.

Keywords: Monitoring; Evaluation; Data Science; LLM; Artificial Intelligence; Competitive Intelligence (search for similar items in EconPapers)
Date: 2025-03-13
References: Add references at CitEc
Citations:

Published in Benoît Epron; Evelyne Broudoux; Ghislaine Chartron. Information et intelligence artificielle, De Boeck Supérieur, pp.147-159, 2025, 978-2-8073-6987-0. ⟨10.3917/dbu.broud.2025.01.0147⟩

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hal:journl:halshs-05474783

DOI: 10.3917/dbu.broud.2025.01.0147

Access Statistics for this paper

More papers in Post-Print from HAL
Bibliographic data for series maintained by CCSD ().