Geometry of Learning in Multilayer Perceptrons

Shun-ichi Amari (), Hyeyoung Park () and Tomoko Ozeki ()
Additional contact information
Shun-ichi Amari: RIKEN, Brain Science Institute
Hyeyoung Park: Kyungpook National University, Dept. of Computer Science, College of Natural Science
Tomoko Ozeki: RIKEN, Brain Science Institute

A chapter in COMPSTAT 2004 — Proceedings in Computational Statistics, 2004, pp 49-60 from Springer

Abstract: Abstract Neural networks provide a good model of learning from statistical data. Multilayer perceptron is regarded as a statistical model in which a nonlinear input-output relation is realized. The set of multilayer perceptrons forms a statistical manifold in which learning and estimation takes place. This is a Riemannian manifold with Fisher information metric. However, such a hierarchical model includes algebraic singularities at which the Fisher information matrix degenerates. This causes various difficulties in learning and statistical estimation. The present paper elucidates the structure of singularities, and how they influence the behavior of learning. The paper describes a new learning algorithm, named the natural gradient method, to overcome such difficulties. Various statistical problems in singular models are discussed, and the models selection criteria (AIC and MDL) are studied in this framework.

Keywords: Learning; information geometry; singular statistical model; neural networks (search for similar items in EconPapers)
Date: 2004
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:sprchp:978-3-7908-2656-2_3

Ordering information: This item can be ordered from
http://www.springer.com/9783790826562

DOI: 10.1007/978-3-7908-2656-2_3

Access Statistics for this chapter

More chapters in Springer Books from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().