Large Language Models for Statistical Analysis: Can they Replace Domain-Specific Software Packages?

David Ajayi

No zj5pc_v1, MetaArXiv from Center for Open Science

Abstract: Large language models (LLMs) represent one of the most significant advances in artificial intelligence in recent decades. Although LLMs are widely used to generate statistical code in domain-specific languages such as R, Python, and SAS, they are increasingly employed to perform data analysis directly. Prior studies have examined the use of LLMs in statistical analysis, but important gaps remain, including limited evidence on their proficiency in data manipulation and Bayesian statistical modeling. The present study was designed to evaluate the performance of common LLMs across a wide range of data analysis tasks, including data reading, data manipulation, descriptive statistics, contingency table analysis, mean comparison tests, correlation analysis, regression modeling, and Bayesian inference. Six large language models were assessed: ChatGPT 5.3, Gemini 3.1, Claude Sonnet 4.6, Microsoft Copilot GPT 5.1, Grok 4.2, and DeepSeek 3.2. All models were tested using their free-tier access, except for ChatGPT, which was evaluated through a paid subscription. Fully or partially simulated datasets were used in this study, and strict scoring criteria were implemented, in that the outputs from the LLMs must be consistent with those from R, and they must be reproducible upon re-run. Gemini, ChatGPT, and Claude achieved 100% accuracy in data reading and descriptive statistics. Gemini and Claude generated correct results for mean comparison tests. ChatGPT and Claude produced accurate outputs in correlation and regression analyses. None of the LLMs achieved 100% accuracy in data manipulation, contingency table analyses, and Bayesian modeling. On average, no LLM achieved perfect accuracy. The overall performance of Gemini, ChatGPT, and Claude was comparable, whereas Grok, Copilot, and DeepSeek performed poorly. Limitations in data manipulation and some inferential statistical methods suggest that LLMs cannot yet replace domain-specific software packages. Therefore, LLMs are better suited as complementary tools rather than standalone applications for rigorous statistical analysis.

Date: 2026-05-21
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6a0ca29f34406a10992c36db/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:zj5pc_v1

DOI: 10.31219/osf.io/zj5pc_v1

Access Statistics for this paper

More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().