The Evaluation of GenAI Capabilities to Implement Professional Tasks

Yaroslav Kouzminov () and Ekaterina Kruchinskaia ()
Additional contact information
Yaroslav Kouzminov: National Research University Higher School of Economics
Ekaterina Kruchinskaia: National Research University Higher School of Economics

Foresight and STI Governance (Foresight-Russia till No. 3/2015), 2024, vol. 18, issue 4, 67-76

Abstract: Generative AI (GenAI) or large language models (LLMs) have been running the world since 2022, but despite all the trends surrounding the use of generative models, these cannot yet be used professionally. While they are most valued for knowing everything, nonetheless GenAI models cannot explain and prove. In this way we conceptualize the most recent problem of LLMs as the general trend of mistakes even in the core of knowledge and non-causality of mistake via the complexity of question, as the mistake can be named as an accident and be everywhere as the most limitation of professionalism. At their current stage of development, LLMs are not widely used in a professional context, nor have they replaced human workers. They do not event extend workers professional abilities. These limitations of GenAI have one general: non-repayment. This article seeks to analyze GenAI professional viability by examining two models (GigaChatPro, GPT-4) in three fields of knowledge (economics, law, education) based on our unique Bloom taxonomy benchmark. To prove our assumption concerning the low possibility of its professional usage, we test three hypotheses: 1) the number of parameters of models have low elasticity regarding difficulty and taxonomy with even the right answer; 2) difficulty and taxonomy jointly have no effect on the correctness of an answer, 3) multiple choice is a factor that decreases the number of right answers of a model. We also present the results of GPT-4 and GigaChat MAX on our benchmark. Finally, we suggest what can be done about the limitations of GenAI’s architecture to reach at least a quasi-professional use.

Keywords: professionalism; generative artificial intelligence; professional use of language models; graphs of knowledge; orchestration; Bloom’s taxonomy. (search for similar items in EconPapers)
JEL-codes: C45 Y80 (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://foresight-journal.hse.ru/article/view/24104/20267 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hig:fsight:v:18:y:2024:i:4:p:67-76

Access Statistics for this article

More articles in Foresight and STI Governance (Foresight-Russia till No. 3/2015) from National Research University Higher School of Economics
Bibliographic data for series maintained by Nataliya Gavrilicheva () and Mikhail Salazkin ().