Empirical calibration of time series monitoring methods using receiver operating characteristic curves

Jacqueline Cohen, Samuel Garman and Wilpen Gorr

International Journal of Forecasting, 2009, vol. 25, issue 3, 484-497

Abstract: Time series monitoring methods, such as the Brown and Trigg methods, have the purpose of detecting pattern breaks (or "signals") in time series data reliably and in a timely fashion. Traditionally, researchers have used the average run length (ARL) statistic on results from generated signal occurrences in simulated time series data to calibrate and evaluate these methods, with a focus on timeliness of signal detection. This paper investigates the receiver operating characteristic (ROC) framework, well-known in the diagnostic decision making literature, as an alternative to ARL analysis for time series monitoring methods. ROC analysis traditionally uses real data to address the inherent tradeoff in signal detection between the true and false positive rates when varying control limits. We illustrate ROC analysis using time series data on crime at the patrol district level in two cities, and use the concept of Pareto frontier ROC curves and reverse functions for methods such as Brown's and Trigg's that have parameters affecting signal-detection performance. We compare the Brown and Trigg methods to three benchmark methods, including one commonly used in practice. The Brown and Trigg methods collapse to the same simple method on the Pareto frontier and dominate the benchmark methods under most conditions. The worst method is the one commonly used in practice.

Keywords: Time; series; monitoring; ROC; curve; Average; run; length; statistic; Exponential; smoothing; Structural; breaks; Step; jumps; Outliers (search for similar items in EconPapers)
Date: 2009
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (12)

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0169-2070(08)00141-6
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:intfor:v:25:y:2009:i:3:p:484-497

Access Statistics for this article

International Journal of Forecasting is currently edited by R. J. Hyndman

More articles in International Journal of Forecasting from Elsevier
Bibliographic data for series maintained by Catherine Liu ().