When Small Wins Big: Classification Tasks Where Compact Models Outperform Original GPT-4

Quand les petits gagnent les grands: tâches de classification pour lesquelles les modèles compacts sont plus performants que les modèles originaux GPT-4

Baptiste Lefort (), Eric Benhamou, Jean-Jacques Ohana, Beatrice Guez, David Saltiel and Damien Challet
Additional contact information
Baptiste Lefort: MICS - Mathématiques et Informatique pour la Complexité et les Systèmes - CentraleSupélec - Université Paris-Saclay, A.I. For Alpha
Eric Benhamou: A.I. For Alpha
Jean-Jacques Ohana: A.I. For Alpha
Beatrice Guez: A.I. For Alpha
David Saltiel: A.I. For Alpha

Working Papers from HAL

Abstract: This paper evaluates Large Language Models (LLMs) on financial text classification, comparing GPT-4 (1.76 trillion parameters) against FinBERT (110 million parameters) and FinDROBERTA (82.1 million parameters). We achieved a classification task on short financial sentences involving multiple divergent insights with both textual and numerical data. We developed a market-based large dataset that enabled us to fine-tune the models on a real-world ground truth. Utilizing a marketbased dataset for fine-tuning on extensive datasets, we achieved significant enhancements with Fin-BERT and FinDROBERTA over GPT-4. However, the use of a bagging majority classifier did not yield performance improvements, demonstrating that the principles of Condorcet's jury Theorem do not apply, suggesting a lack of independence among the models and similar behavior patterns across all evaluated models. Our results indicate that for complex sentiment classification, compact models match larger models, even with fine-tuning. The fine-tuned models are made available as opensource for additional research.

Date: 2024-10-16
Note: View the original document on HAL open archive server: https://hal.science/hal-04739931v1
References: Add references at CitEc
Citations:

Downloads: (external link)
https://hal.science/hal-04739931v1/document (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hal:wpaper:hal-04739931

Access Statistics for this paper

More papers in Working Papers from HAL
Bibliographic data for series maintained by CCSD ().