Mapping model validation metrics to subject matter expert scores for model adequacy assessment

Kirubel Teferra, Michael D. Shields, Adam Hapij and Raymond P. Daddazio

Reliability Engineering and System Safety, 2014, vol. 132, issue C, 9-19

Abstract: This paper develops a novel approach to incorporate the contributions of both quantitative validation metrics and qualitative subject matter expert (SME) evaluation criteria in model validation assessment. The relationship between validation metrics (input) and SME scores (output) is formulated as a classification problem, and a probabilistic neural network (PNN) is constructed to execute this mapping. Establishing PNN classifiers for a wide variety of combinations of validation metrics allows for a quantitative comparison of validation metric performance in representing SME judgment. An advantage to this approach is that it semi-automates the model validation process and subsequently is capable of incorporating the contributions of large data sets of disparate response quantities of interest in model validation assessment. The effectiveness of this approach is demonstrated on a complex real-world problem involving the shock qualification testing of a floating shock platform. A data set of experimental and simulated pairs of time history comparisons along with associated SME scores and computed validation metrics is obtained and utilized to construct the PNN classifiers through K-fold cross validation. A wide range of validation metrics for time history comparisons is considered including feature-specific metrics (phase and magnitude error), a frequency metric (shock response spectra), a time-frequency metric (wavelet decomposition), and a global metric (index of agreement). The PNN classifiers constructed using a Parzen kernel for the class conditional probability density function whose smoothing parameter is optimized using a genetic algorithm performs well in representing SME judgment.

Keywords: Model validation; Validation metrics; Subject matter experts; Probabilistic neural networks; Machine learning (search for similar items in EconPapers)
Date: 2014
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0951832014001653
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:reensy:v:132:y:2014:i:c:p:9-19

DOI: 10.1016/j.ress.2014.07.010

Access Statistics for this article

Reliability Engineering and System Safety is currently edited by Carlos Guedes Soares

More articles in Reliability Engineering and System Safety from Elsevier
Bibliographic data for series maintained by Catherine Liu ().