Monitoring the data quality of data streams using a two-step control scheme

Miaomiao Yu, Chunjie Wu and Fugee Tsung

IISE Transactions, 2019, vol. 51, issue 9, 985-998

Abstract: Data-rich environments provide unprecedented opportunities for monitoring data quality. This article focuses on the quality of data streams. We use indicator variables to measure the six dimensions of data quality and a glitch index to indicate the poor level of quality. A two-step control scheme is proposed considering two relationships: the inter- and intra-correlation. In the first step, the Mahalanobis distance is applied to an χ2-type control chart to monitor the quality of a data stream. In the second step, a Shewhart control chart is built based on a weighted-sum statistic, which measures the quality of the whole process. The feasibility and effectiveness of the control scheme are illustrated through detailed simulation studies and one landslide example. The simulated results, considering the three cases of no correlation, low correlation, and high correlation, show that the proposed approach can detect the mean shift in multi-attribute data sensitively and robustly. The example, in which sensors are used to collect data on accelerations in Taiwan, demonstrates the superiority of our design over four traditional control charts, producing the closest type-I error to the given level and the highest power under the same type-I error.

Date: 2019
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/24725854.2018.1530487 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:uiiexx:v:51:y:2019:i:9:p:985-998

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/uiie20

DOI: 10.1080/24725854.2018.1530487

Access Statistics for this article

IISE Transactions is currently edited by Jianjun Shi

More articles in IISE Transactions from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().