Automated Test Generation Using Large Language Models

Marcin Andrzejewski, Nina Dubicka, Jędrzej Podolak, Marek Kowal and Jakub Siłka ()
Additional contact information
Marcin Andrzejewski: GenerativeAI Academic Research Team (GART), Capgemini Insights & Data, 54-202 Wroclaw, Poland
Nina Dubicka: GenerativeAI Academic Research Team (GART), Capgemini Insights & Data, 54-202 Wroclaw, Poland
Jędrzej Podolak: GenerativeAI Academic Research Team (GART), Capgemini Insights & Data, 54-202 Wroclaw, Poland
Marek Kowal: GenerativeAI Academic Research Team (GART), Capgemini Insights & Data, 54-202 Wroclaw, Poland
Jakub Siłka: Faculty of Applied Mathematics, Silesian University of Technology, 44-100 Gliwice, Poland

Data, 2025, vol. 10, issue 10, 1-20

Abstract: This study explores the potential of generative AI, specifically Large Language Models (LLMs), in automating unit test generation in Python 3.13. We analyze tests, both those created by programmers and those generated by LLM models, for fifty source code cases. Our main focus is on how the choice of model, the difficulty of the source code, and the prompting strategy influence the quality of the generated tests. The results show that AI models can help automate test creation for simple code, but their effectiveness decreases for more complex tasks. We introduce an embedding-based similarity analysis to assess how closely AI-generated tests resemble human-written ones, revealing that AI outputs often lack semantic diversity. The study also highlights the potential of AI models for rapid test prototyping, which can significantly speed up the software development cycle. However, further customization and training of the models on specific use cases is needed to achieve greater precision. Our findings provide practical insights into integrating LLMs into software testing workflows and emphasize the importance of prompt design and model selection.

Keywords: GenAI; TDD; embeddings; code coverage; LLM (search for similar items in EconPapers)
JEL-codes: C8 C80 C81 C82 C83 (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2306-5729/10/10/156/pdf (application/pdf)
https://www.mdpi.com/2306-5729/10/10/156/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jdataj:v:10:y:2025:i:10:p:156-:d:1761793

Access Statistics for this article

Data is currently edited by Ms. Becky Zhang

More articles in Data from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().