Simulation Model Validation With Selective Metric Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods lack an effective and efficient way to validate the quality of simulation models, particularly in complex systems like autonomous vehicles, which is crucial for ensuring their reliability and safety.
Innovation Solution
A computer-implemented method that receives simulation and measurement data, applies multiple metrics, selects relevant metrics based on predefined conditions, and determines a validation metric through optimization, reducing the need for physical tests and enabling scalable validation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple predetermined metrics are applied to simulation and measurement data, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent extracts and selects only the most relevant metrics from a comprehensive set of predetermined metrics based on predefined conditions and domain knowledge. This extraction process removes unnecessary metrics that would otherwise complicate the validation system while preserving the essential measurement capabilities needed for accurate model validation.
Solution Approach 2:
The patent applies different metrics selectively based on the specific validation context, data characteristics, and model type. Rather than uniformly applying all metrics, the system tailors the metric selection to local requirements, optimizing the balance between measurement precision and system complexity for each validation scenario.
2Reliability
If comprehensive validation metrics are used, then reliability is improved, but loss of time increases
Solution Approach 1:
The patent performs preliminary filtering and selection of metrics based on predefined conditions before the actual validation computation. By pre-identifying relevant metrics and establishing selection criteria in advance, the system avoids unnecessary computations during the validation process, thereby maintaining high reliability while reducing validation time.
Solution Approach 2:
The patent applies a subset of the most critical metrics rather than all available metrics. This partial action approach focuses computational resources on the most impactful validation measures, achieving sufficient reliability for practical purposes while significantly reducing the time required for complete validation.
3Manufacturing precision
If multiple metrics are applied to ensure model quality, then manufacturing precision is improved, but ease of manufacture deteriorates
Solution Approach 1:
The patent develops a universal metric selection framework that can be applied across different simulation domains and model types. The predefined conditions and selection criteria are designed to be domain-agnostic, allowing the same systematic approach to be used for various validation scenarios, thereby improving manufacturability while maintaining precision through consistent application of the framework.
Solution Approach 2:
The patent adjusts the selection of metrics and predefined conditions based on specific validation requirements and data characteristics. By dynamically changing which metrics are applied and how they are weighted, the system maintains high manufacturing precision while adapting to different implementation contexts, making the validation system easier to manufacture for specific applications.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
A general aspect of the present disclosure relates to a computer-implemented method for determining a validation metric for determining model quality. The method includes receiving a first data series comprising results of a simulation, receiving a second data series comprising results of a measurement, receiving visual validation of the first data series based on a comparison to the second data series to obtain an assessment of the first data series, applying a plurality of predetermined metrics to the first data series and the second data series and selecting one or more metrics from the plurality of metrics based on a predetermined condition, and determining a validation metric based on optimization of parameters using the selected one or more metrics and the assessment of the first data series.