Knowledge-based Model Validation Using a Custom Metric

Vehicle models have a long history of research and, as of today, are able to model the involved physics in a reasonable manner. However, each new vehicle has its own unique characteristics or parameters. Identifying these is the main task of an engineer. Validating whether the correct parameter set has been chosen is a tedious task and often can only be performed by experts. Metrics commonly used in literature can compare different results under certain aspects. However, they fail to answer the question: Are the models accurate enough? In this article, we propose the usage of a custom metric trained on expert knowledge to tackle this problem. Our approach involves three main steps: first, the formalized collection of subject matter experts' opinions on the question: Having seen the measurement and simulation time-series in comparison, is the model quality sufficient? From this step, we obtain a dataset that quantifies the sufficiency of a simulation result based on a comparison to corresponding experimental data. In the second step, we compute common model metrics on the measurement and simulation time-series and use these model metrics as features in a regression model. Third, we fit a regression model to the experts' opinions. This regression model, i.e., our custom metric, can then predict the sufficiency of a new simulation result and provide a confidence level on this prediction.