A Comparison of Large Language Models and Genetic Programming for Program Synthesis

Dominik Sobania (), Justyna Petke (), Martin Briesch () and Franz Rothlauf ()
Additional contact information
Dominik Sobania: Johannes-Gutenberg University, Germany
Justyna Petke: University College London, United Kingdom
Martin Briesch: Johannes-Gutenberg University, Germany
Franz Rothlauf: Johannes-Gutenberg University, Germany

No 2414, Working Papers from Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz

Abstract: Large language models have recently become known for their ability to generate computer programs, especially through tools such as GitHub Copilot, a domain where genetic programming has been very successful so far. Although they require different inputs (free-text vs. input/output examples) their goal is the same – program synthesis. Therefore, in this work we compare how well GitHub Copilot and genetic programming perform on common program synthesis benchmark problems. We study the structure and diversity of the generated programs by using well-known software metrics. We find that GitHub Copilot and genetic programming solve a similar number of benchmark problems (85.2% vs. 77.8%, respectively). We find that GitHub Copilot generated smaller and less complex programs as genetic programming, while genetic programming is able to find new and unique problem solving strategies. This increase in diversity of solutions comes at a cost. When analyzing the success rates for 100 runs per problem, GitHub Copilot outperforms genetic programming on over 50% of the problems.

Keywords: Program Synthesis; Genetic Programming; Large Language Models; Codex; GitHub Copilot; Software Engineering. (search for similar items in EconPapers)
Pages: 13 pages
Date: 2024-08-08
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://download.uni-mainz.de/RePEc/pdf/Discussion_Paper_2414.pdf First version, 2024 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:jgu:wpaper:2414

Access Statistics for this paper

More papers in Working Papers from Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz Contact information at EDIRC.
Bibliographic data for series maintained by Research Unit IPP ().