Asymptotic Analysis of the Bias–Variance Trade-Off in Subsampling Metropolis–Hastings

Shuang Liu ()
Additional contact information
Shuang Liu: Department of Statistics and Data Science, School of Economics, Xiamen University, Xiamen 361005, China

Mathematics, 2025, vol. 13, issue 21, 1-30

Abstract: Markov chain Monte Carlo (MCMC) methods are fundamental to Bayesian inference but are often computationally prohibitive for large datasets, as the full likelihood must be evaluated at each iteration. Subsampling-based approximate Metropolis–Hastings (MH) algorithms offer a popular alternative, trading a manageable bias for a significant reduction in per-iteration cost. While this bias–variance trade-off is empirically understood, a formal theoretical framework for its optimization has been lacking. Our work establishes such a framework by bounding the mean squared error (MSE) as a function of the subsample size ( m ), the data size ( n ), and the number of epochs ( E ). This analysis reveals two optimal asymptotic scaling laws: the optimal subsample size is m ★ = O ( E 1 / 2 ) , leading to a minimal MSE that scales as MSE ★ = O ( E − 1 / 2 ) . Furthermore, leveraging the large-sample asymptotic properties of the posterior, we show that when augmented with a control variate, the approximate MH algorithm can be asymptotically more efficient than the standard MH method under ideal conditions. Experimentally, we first validate the two optimal asymptotic scaling laws. We then use Bayesian logistic regression and Softmax classification models to highlight a key difference in convergence behavior: the exact algorithm starts with a high MSE that gradually decreases as the number of epochs increases. In contrast, the approximate algorithm with a practical control variate maintains a consistently low MSE that is largely insensitive to the number of epochs.

Keywords: approximate MCMC; bias–variance trade-off; scaling laws; subsampling; variance reduction (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2025
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/21/3395/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/21/3395/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2025:i:21:p:3395-:d:1779063

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().