Use of Static Surrogates in Hyperparameter Optimization

Dounia Lakhmiri () and Sébastien Digabel
Additional contact information
Dounia Lakhmiri: Polytechnique Montréal GERAD
Sébastien Digabel: Polytechnique Montréal GERAD

SN Operations Research Forum, 2022, vol. 3, issue 1, 1-18

Abstract: Abstract Optimizing the hyperparameters and architecture of a neural network is a long yet necessary phase in most applications. This consuming process can benefit from strategies designed to discard low-quality configurations and quickly focus on more promising candidates. This work aims at enhancing HyperNOMAD, a library that adapts a direct search derivative-free optimization algorithm to tune both the architecture and the training of a neural network simultaneously. Two static surrogates are developed to trigger an early stopping during the configuration evaluation and strategically rank a pool of candidates. These additions to HyperNOMAD are shown to reduce its resource consumption by orders of magnitude without harming the quality of the proposed solutions.

Keywords: Hyperparameter optimization (HPO); Derivative-free optimization (DFO); Blackbox optimization (BBO) (search for similar items in EconPapers)
Date: 2022
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s43069-022-00128-w Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:snopef:v:3:y:2022:i:1:d:10.1007_s43069-022-00128-w

Ordering information: This journal article can be ordered from
https://www.springer.com/journal/43069

DOI: 10.1007/s43069-022-00128-w

Access Statistics for this article

SN Operations Research Forum is currently edited by Marco Lübbecke

More articles in SN Operations Research Forum from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().