Bias, Calibration and Stability in Logistic Regression Models: A Comparative Simulation Study of MLE, Firth and Ridge Methods

Ngetich Festus and Dr. Apaka Rangita
Additional contact information
Ngetich Festus: Department of Statistics and Actuarial Science, Maseno University, Kenya
Dr. Apaka Rangita: N/A

International Journal of Research and Innovation in Applied Science, 2025, vol. 10, issue 7, 838-852

Abstract: Logistic regression is widely used to model binary outcomes but the traditional maximum likelihood estimation performs poorly in small samples, rare events and in cases of correlated predictors. This simulation study compared the MLE, the Firth’s bias-reduced and the ridge-penalized logistic regression across diverse sample sizes (n = 20, 100, 1000), event rates (5%, 20%, 50%) and predictor correlations (Ï = 0.1, 0.5, 0.8). The key performance metrics were estimation bias, calibration slope and bootstrap-based coefficient variability. Results show that MLE suffers extreme bias and instability in small samples with rare events but recovers well at n = 1000 thus achieving nearly perfect calibration with slope approximately 1. The Firth’s method mitigates bias and complete-separation issues in small samples though it also introduces severe calibration distortion (slopes >50). Ridge regression on the other hand provide the most stable coefficient estimates from the bootstrap SDs significantly lower than those of MLE but shows inconsistent calibration especially under sparse conditions. Overall, the Firth is recommended for inference in sparse data, the ridge for prediction in high-dimensional and multicollinear settings and the MLE for large and well-powered datasets. This study demonstrates the significance of aligning estimation methods with those of data characteristics to ensure accuracy and robustness of logistic regression modeling.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.rsisinternational.org/journals/ijrias/ ... -issue-7/838-852.pdf (application/pdf)
https://rsisinternational.org/journals/ijrias/arti ... h-and-ridge-methods/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bjf:journl:v:10:y:2025:i:7:p:838-852

Access Statistics for this article

International Journal of Research and Innovation in Applied Science is currently edited by Dr. Renu Malsaria

More articles in International Journal of Research and Innovation in Applied Science from International Journal of Research and Innovation in Applied Science (IJRIAS)
Bibliographic data for series maintained by Dr. Renu Malsaria ().