Chrome Extension
WeChat Mini Program
Use on ChatGLM

Certifiable Evaluation for Autonomous Vehicle Perception Systems using Deep Importance Sampling (Deep IS)

International Conference on Intelligent Transportation Systems (ITSC)(2022)

Cited 0|Views23
No score
Abstract
Evaluating the performance of autonomous vehicles (AV) and their complex AI-driven functionalities to high precision under naturalistic conditions remains a challenge, especially when the failure or dangerous cases are rare. Rarity does not only require an enormous sample size for a naive method to achieve high confidence residual risk estimation, but it can also cause serious risk underestimation issues that is hard to detect. Meanwhile, the state-of-the-art rare safety-critical event evaluation approach that comes with a correctness guarantee can compute an upper bound for the true risk under certain conditions, which limits its practical uses. In this work, we propose Deep Importance Sampling (Deep IS) framework that utilizes a deep neural network to obtain an efficient less biased risk estimate, with an efficiency that is on par with that of the state-of-the-art method. In the numerical experiment evaluating the misclassification rate of a traffic sign classifier, Deep IS only needs 1/40-th of the samples required by a naive sampling method to achieve 10% relative error. Furthermore, the estimate produced by Deep IS is 10 times less conservative compared to the risk upper bound and only off by at most 10% difference to the true target. This efficient deep-learning-based IS procedure promises a highly efficient method to deal with often high-dimensional functional safety problems with rare naturalistic failure cases that are prevalent in AV domains.
More
Translated text
Key words
naive method,high confidence residual risk estimation,serious risk,state-of-the-art rare safety-critical event evaluation approach,Deep Importance Sampling,deep neural network,efficient less biased risk estimate,naive sampling method,risk upper bound,high-dimensional functional safety problems,rare naturalistic failure cases,certifiable evaluation,autonomous vehicle perception systems,autonomous vehicles,AI-driven functionalities,naturalistic conditions,enormous sample size
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined