SKR1: Benchmark for Testing Knowledge About Slovak Realia for Large Language Models

Marek Dobeš

Acta Informatica Pragensia, vol. preprint

Abstract: Background: To objectively evaluate the capabilities of large language models (LLMs), we need to develop tools that enable such assessment. While numerous benchmarks exist, the vast majority are in English and focus on general knowledge, often overlooking the cultural and factual specifics of smaller countries.Objective: Currently, there is no benchmark that tests LLMs΄ knowledge of Slovak realia. At the same time, LLM performance in this domain remains inadequate. To objectively measure and compare these capabilities, our goal is to develop and validate a specialized benchmark for assessing LLMs΄ knowledge of Slovak cultural and factual context.Methods: We created a set of 35 questions on Slovak culture, geography, history and language. We designed them to provide unambiguous answers suitable for automated evaluation. Subsequently, we presented the questions to three major language models-DeepSeek V3, OpenAI GPT-4o and Llama 3.Results: DeepSeek scored 54% of correct answers, OpenAI GPT scored 51% and Llama scored 40%. The models scored best in geography questions. Overall scores show that models are not very good in recognising Slovak realia.Conclusion: We present the benchmark for evaluating large language models on Slovak-related knowledge. Even the most advanced current models, including OpenAI GPT and DeepSeek, answered only around half of the questions correctly. This highlights a significant gap in international LLMs΄ understanding of culturally specific facts, underscoring the need for specialized, nationally tailored language models.

Keywords: LLM; Benchmark; Slovak realia; DeepSeek; OpenAI GPT; Llama (search for similar items in EconPapers)
References: Add references at CitEc
Citations:

Downloads: (external link)
http://aip.vse.cz/doi/10.18267/j.aip.300.html (text/html)
free of charge

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:prg:jnlaip:v:preprint:id:300

Ordering information: This journal article can be ordered from
Redakce Acta Informatica Pragensia, Katedra systémové analýzy, Vysoká škola ekonomická v Praze, nám. W. Churchilla 4, 130 67 Praha 3
http://aip.vse.cz

DOI: 10.18267/j.aip.300

Access Statistics for this article

Acta Informatica Pragensia is currently edited by Editorial Office

More articles in Acta Informatica Pragensia from Prague University of Economics and Business Contact information at EDIRC.
Bibliographic data for series maintained by Stanislav Vojir ().