A Bayesian hierarchical model with spatially varying dispersion for reference-free cell type deconvolution in spatial transcriptomics

Xuan Li, Yincai Tang, Jingsi Ming and Xingjie Shi

Statistical Theory and Related Fields, 2025, vol. 9, issue 2, 178-212

Abstract: A major challenge in spatial transcriptomics (ST) is resolving cellular composition, especially in technologies lacking single-cell resolution. The mixture of transcriptional signals within spatial spots complicates deconvolution and downstream analyses. To uncover the spatial heterogeneity of tissues, we introduce SvdRFCTD, a reference-free spatial transcriptomics deconvolution method, which estimates the cell type proportions at each spot on the tissue. To fully capture the heterogeneity in the ST data, we combine SvdRFCTD with a Bayesian hierarchical negative binomial model with spatial effects incorporated in both the mean and dispersion of the gene expression, which is used to explicitly model the generative mechanism of cell type proportions. By integrating spatial information and leveraging marker gene information, SvdRFCTD accurately estimates cell type proportions and uncovers complex spatial patterns. We demonstrate the ability of SvdRFCTD to identify cell types on simulated datasets. By applying SvdRFCTD to mouse brain and human pancreatic ductal adenocarcinomas datasets, we observe significant cellular heterogeneity within the tissue sections and successfully identify regions with high proportions of aggregated cell types, along with the spatial relationships between different cell types.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/24754269.2025.2495651 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:tstfxx:v:9:y:2025:i:2:p:178-212

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/tstf20

DOI: 10.1080/24754269.2025.2495651

Access Statistics for this article

Statistical Theory and Related Fields is currently edited by Zhao Wei

More articles in Statistical Theory and Related Fields from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().