From Concepts to Model: Automating Feature Extraction of Agent-Based Models Using Large Language Models

Siamak Khatami () and Christopher Konstantin Frantz ()
Additional contact information
Siamak Khatami: https://www.ntnu.edu/employees/siamak.khatami
Christopher Konstantin Frantz: https://christopherfrantz.org/

Journal of Artificial Societies and Social Simulation, 2025, vol. 28, issue 3, 9

Abstract: Converting conceptual models into simulation models is a primary challenge in the Agent-based Modeling (ABM) development lifecycle, often acting as a bottleneck due to communication gaps among different stakeholders, particularly programmers and mathematicians. To address this issue, Large Language Models (LLMs) and, more specifically, Question-answering (QA) models (also known as Conversational Artificial Intelligence (CAI)) can play a central role. However, using QA models and related Natural Language Processing (NLP) techniques for auto-generating simulation models in an integrated process is promising with respect to increasing efficiency, consistency, and, potentially, quality of the extracted conceptual model. Drawing on contemporary QA models, our proposed approach involves the systematic extraction of model features from descriptions to generate a conceptual model that is the precursor for automating the generation of model implementations. A central contribution of this work is to establish a systematic method for this initial step, extracting simulation-relevant information from the conceptual model and presenting it in a generic machine- and human-readable format suitable for future applications facilitating automated code generation, taking into account the continuous technological progression in this area. To this end, this article introduces a baseline schema of information pertinent to developing basic conceptual ABMs and employs a testbed methodology to examine the performance of a wide range of contemporary LLMs like Llama 2 and 3 embedded in QA models, alongside available commercial QAs (like ChatGPT-3.5 and ChatGPT-4o). The evaluation relies on a combination of automated cosine similarity and manual similarity assessment to establish generic metrics to assess QA model performance. The results of contemporary models show the dominant performance of commercially available models (ChatGPT-4o) while highlighting the increasing potential for the use of open-source LLMs (e.g., Llama 2 and 3) for the purpose of model feature extraction. We discuss the associated implications for automated code generation as well as future directions while also exploring the broader potential for the use of QA models to support other stages of the development cycle of ABMs.

Keywords: Model Extraction; Artificial Intelligence; Natural Language Processing; Large Language Models; Question Answering; Agent-Based Modeling (search for similar items in EconPapers)
Date: 2025-06-30
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.jasss.org/28/3/9/9.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:jas:jasssj:2024-115-3

Access Statistics for this article

More articles in Journal of Artificial Societies and Social Simulation from Journal of Artificial Societies and Social Simulation
Bibliographic data for series maintained by Francesco Renzini ().