Validation of Trace-Driven Simulation Models: A Novel Regression Test

Jack Kleijnen (), Bert Bettonvil and Willem Groenendaal ()
Additional contact information
Bert Bettonvil: Department of Information Systems and Auditing (BIKA)/Center for Economic Research, Tilburg University, 5000 LE Tilburg, The Netherlands

Management Science, 1998, vol. 44, issue 6, 812-819

Abstract: This paper argues that it is wrong to require that regressing the outputs of a trace-driven simulation on the observed real outcomes should give a 45° (unit slope) line through the origin (zero intercept). This note proposes instead an alternative requirement: the responses of the simulated and the real systems should have the same means and the same variances. To test statistically whether this requirement is satisfied, a novel procedure is derived: regress the differences between simulated and real responses on their associated sums, and test whether the resulting intercept and slope are both zero. This novel but simple test assumes identically, independently, and normally distributed outputs of the real system and the simulated system. The old and the new procedures are investigated in extensive Monte Carlo experiments that simulate M/M/1 queueing systems. The conclusions are: (i) the naive intuitive test rejects a valid simulation model substantially more often than the novel test does; (ii) the naive test shows "perverse" behavior within a certain domain: the worse the simulation model, the higher its estimated probability of acceptance; and (iii) the novel test does not reject a valid simulation model too often (its type I error probability is correct), provided the queueing response is transformed appropriately to obtain (nearly) normally distributed responses.

Keywords: Correlation; Paired Observations; Goodness-of-Fit; Type II Error; Power; Simultaneous Tests (search for similar items in EconPapers)
Date: 1998
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (15)

Downloads: (external link)
http://dx.doi.org/10.1287/mnsc.44.6.812 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:44:y:1998:i:6:p:812-819

Access Statistics for this article

More articles in Management Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().