How Effectively Train Large-Scale Machine Learning Models?

Aven Samareh () and Mahshid Salemi Parizi ()
Additional contact information
Aven Samareh: University of Washington
Mahshid Salemi Parizi: University of Washington

A chapter in Optimization in Large Scale Problems, 2019, pp 97-110 from Springer

Abstract: Abstract The stochastic gradient method (SGM)Stochastic gradient method (SGM) is widely used as an optimization tool in many machine learning applications including support vector machines (SVM)s, Support vector machines (SVM) logistic regression, graphical models and deep learning. SGM computes the estimates of the gradient from a single randomly chosen sample in each iteration. Therefore, applying a stochastic gradient method for large-scale machine learning problems can be computationally efficient. In this work, we focus on generating generalization bounds for a randomized algorithm Algorithm such as Random Fourier features learned with stochastic gradient descent algorithm. Our findings are based on a mutual relationship between the generalization error of an algorithm and its stability Stability properties. The stability of an algorithm is measured by the generalization error Generalization error regarding the absolute difference between the testing and the training error. Training error Overall, an algorithm is called stable if by changing any single training data Training data point the training error varies slightly. In this work, we measured the stability of stochastic gradient method (SGM) for learning an approximated Fourier primal support vector machine. In particular, under certain regularity assumptions, we showed that a randomized algorithm such as Random Fourier feature where is trained by a stochastic gradient method (SGM) with few iterations has vanishing generalization error. Therefore, the iterative optimization algorithm can stop long before its convergence to reduce computational cost. We empirically verified the theoretical findings for different parameters using several data sets.

Keywords: Convex optimization; Convex optimization Random fourier features; Random Fourier Features Support vector machine; Generalization error (search for similar items in EconPapers)
Date: 2019
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:spochp:978-3-030-28565-4_14

Ordering information: This item can be ordered from
http://www.springer.com/9783030285654

DOI: 10.1007/978-3-030-28565-4_14

Access Statistics for this chapter

More chapters in Springer Optimization and Its Applications from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().