Optimization Under Uncertainty Explains Empirical Success of Deep Learning Heuristics

Vladik Kreinovich () and Olga Kosheleva ()
Additional contact information
Vladik Kreinovich: University of Texas at El Paso
Olga Kosheleva: University of Texas at El Paso

A chapter in Black Box Optimization, Machine Learning, and No-Free Lunch Theorems, 2021, pp 195-220 from Springer

Abstract: Abstract One of the main objectives of science and engineering is to predict the future state of the world and come up with devices and strategies that would make this future state better. In some practical situations, we know how the state changes with time—e.g., in meteorology, we know the partial differential equations that describe the atmospheric processes. In such situations, prediction becomes a purely computational problem. In many other situations, however, we do not know the equation describing the system’s dynamics. In such situations, we need to learn this dynamics from data. At present, the most efficient way of such learning is to use deep learning—training a neural network with a large number of layers. To make this idea truly efficient, several trial-and-error-based heuristics were discovered, such as the use of rectified linear neurons, softmax, etc. In this chapter, we show that the empirical success of many of these heuristics can be explained by optimization-under-uncertainty techniques.

Date: 2021
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:spochp:978-3-030-66515-9_8

Ordering information: This item can be ordered from
http://www.springer.com/9783030665159

DOI: 10.1007/978-3-030-66515-9_8

Access Statistics for this chapter

More chapters in Springer Optimization and Its Applications from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().