Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles

Jie Min, Yili Hong, Caleb B. King and William Q. Meeker

Journal of the Royal Statistical Society Series C, 2022, vol. 71, issue 4, 987-1013

Abstract: Artificial intelligence (AI) systems have become increasingly common and the trend will continue. Examples of AI systems include autonomous vehicles (AV), computer vision, natural language processing and AI medical experts. To allow for safe and effective deployment of AI systems, the reliability of such systems needs to be assessed. Traditionally, reliability assessment is based on reliability test data and the subsequent statistical modelling and analysis. The availability of reliability data for AI systems, however, is limited because such data are typically sensitive and proprietary. The California Department of Motor Vehicles (DMV) oversees and regulates an AV testing program, in which many AV manufacturers are conducting AV road tests. Manufacturers participating in the program are required to report recurrent disengagement events to California DMV. This information is being made available to the public. In this paper, we use recurrent disengagement events as a representation of the reliability of the AI system in AV, and propose a statistical framework for modelling and analysing the recurrent events data from AV driving tests. We use traditional parametric models in software reliability and propose a new non‐parametric model based on monotonic splines to describe the event process and to estimate the cumulative baseline intensity function of the event process. We develop inference procedures for selecting the best models, quantifying uncertainty and testing heterogeneity in the event process. We then analyse the recurrent events data from four AV manufacturers, and make inferences on the reliability of the AI systems in AV. We also describe how the proposed analysis can be applied to assess the reliability of other AI systems. This paper has online supplementary materials.

Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1111/rssc.12564

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:jorssc:v:71:y:2022:i:4:p:987-1013

Ordering information: This journal article can be ordered from
http://ordering.onli ... 1111/(ISSN)1467-9876

Access Statistics for this article

Journal of the Royal Statistical Society Series C is currently edited by R. Chandler and P. W. F. Smith

More articles in Journal of the Royal Statistical Society Series C from Royal Statistical Society Contact information at EDIRC.
Bibliographic data for series maintained by Wiley Content Delivery ().