Diagnostic Tools for Evaluating and Comparing Simulation-Optimization Algorithms

David J. Eckman (), Shane G. Henderson () and Sara Shashaani ()
Additional contact information
David J. Eckman: Wm Michael Barnes ’64 Department of Industrial and Systems Engineering, Texas A&M University, College Station, Texas 77843
Shane G. Henderson: School of Operations Research and Information Engineering, Cornell University, Ithaca, New York 14853
Sara Shashaani: Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, North Carolina 27695

INFORMS Journal on Computing, 2023, vol. 35, issue 2, 350-367

Abstract: Simulation optimization involves optimizing some objective function that can only be estimated via stochastic simulation. Many important problems can be profitably viewed within this framework. Whereas many solvers—implementations of simulation-optimization algorithms—exist or are in development, comparisons among solvers are not standardized and are often limited in scope. Such comparisons help advance solver development, clarify the relative performance of solvers, and identify classes of problems that defy efficient solution, among many other uses. We develop performance measures and plots, and estimators thereof, to evaluate and compare solvers and diagnose their strengths and weaknesses on a testbed of simulation-optimization problems. We explain the need for two-level simulation in this context and provide supporting convergence theory. We also describe how to use bootstrapping to obtain error estimates for the estimators.

Keywords: analysis of algorithms; simulation; design of experiments; efficiency (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://dx.doi.org/10.1287/ijoc.2022.1261 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:orijoc:v:35:y:2023:i:2:p:350-367

Access Statistics for this article

More articles in INFORMS Journal on Computing from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().