High-dimensional distribution generation through deep neural networks

Dmytro Perekrestenko (), Léandre Eberhard () and Helmut Bölcskei ()
Additional contact information
Dmytro Perekrestenko: Ablacon Inc.
Léandre Eberhard: Upstart Network Inc.
Helmut Bölcskei: ETH Zurich

Partial Differential Equations and Applications, 2021, vol. 2, issue 5, 1-44

Abstract: Abstract We show that every d-dimensional probability distribution of bounded support can be generated through deep ReLU networks out of a 1-dimensional uniform input distribution. What is more, this is possible without incurring a cost—in terms of approximation error measured in Wasserstein-distance—relative to generating the d-dimensional target distribution from d independent random variables. This is enabled by a vast generalization of the space-filling approach discovered in Bailey and Telgarsky (in: Bengio (eds) Advances in neural information processing systems vol 31, pp 6489–6499. Curran Associates, Inc., Red Hook, 2018). The construction we propose elicits the importance of network depth in driving the Wasserstein distance between the target distribution and its neural network approximation to zero. Finally, we find that, for histogram target distributions, the number of bits needed to encode the corresponding generative network equals the fundamental limit for encoding probability distributions as dictated by quantization theory.

Keywords: Deep learning; Neural networks; Generative networks; Space-filling curves; Quantization; Approximation theory; 68T07; 65D15 (search for similar items in EconPapers)
Date: 2021
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s42985-021-00115-6 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:pardea:v:2:y:2021:i:5:d:10.1007_s42985-021-00115-6

Ordering information: This journal article can be ordered from
https://www.springer.com/journal/42985/

DOI: 10.1007/s42985-021-00115-6

Access Statistics for this article

Partial Differential Equations and Applications is currently edited by Zhitao Zhang

More articles in Partial Differential Equations and Applications from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().