Simulation Accuracy Assessment via Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current simulation models for testing technical systems, such as autonomous vehicles, lack confidence in their reliability, limiting their use in clearance decisions due to uncertainty about their accuracy and robustness.
Innovation Solution
A computer-implemented method that uses classification to assess the accuracy and robustness of simulations by comparing simulation results with reference data, providing metrics like confusion matrices and information gain, and training classifiers to predict simulation quality without additional reference measurements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If simulation models are used for testing technical systems, then productivity and cost are improved, but reliability and measurement precision deteriorate due to uncertainty about simulation accuracy
Solution Approach 1:
The patent creates a virtual copy of the real-world test environment through a simulation model. This virtual copy replicates the physical system's behavior, allowing tests to be conducted in silico rather than requiring physical prototypes or real systems. The simulation model serves as a digital twin that can be repeatedly tested without consuming physical resources, thereby improving productivity while maintaining reliability through systematic validation.
Solution Approach 2:
The patent performs preliminary validation of the simulation model by comparing its outputs against reference data from real system tests before using the simulation for actual testing purposes. This preliminary action establishes confidence in the simulation's accuracy and builds a knowledge base that can be used to assess future simulation results, thereby resolving the reliability concern before full-scale productivity benefits are realized.
2Measurement precision
If more reference measurements are conducted to validate simulation accuracy, then measurement precision is improved, but loss of time and productivity decrease
Solution Approach 1:
The patent applies partial validation by comparing simulation results against reference data for a selected subset of test cases rather than exhaustively validating every possible scenario. This partial action is sufficient to establish confidence in the simulation model's general accuracy while avoiding the excessive time consumption that would result from complete validation of all possible test conditions.
Solution Approach 2:
The patent performs preliminary validation using a representative subset of reference data to establish baseline accuracy metrics. Once the simulation model is validated against this preliminary reference set, the same model can be used to assess accuracy for future test scenarios without requiring additional reference measurements, thereby minimizing time loss while maintaining measurement precision.
3Measurement precision
If classification methods are used to assess simulation quality, then device complexity increases, but measurement precision and reliability improve
Solution Approach 1:
The patent introduces a classification system as an intermediary layer between the simulation model and the assessment process. This intermediary automatically compares simulation outputs against reference data, assigns quality classifications, and generates confidence metrics. While this adds some complexity to the assessment system, it dramatically improves measurement precision by providing systematic, objective quality assessment rather than subjective evaluation, and the complexity is justified by the significant improvement in assessment reliability.
Data Source
AI summary
A computer-implemented method for testing a product, in particular software, hardware, or a system comprising hardware and software, in which, depending on input parameters, a simulation of the product is carried out, with the aid of which a particular property of the product is tested. Depending on a comparison between a result of the simulation and a requirement made of the particular property, a first classification is output for the result of the simulation. Depending on a comparison between reference data from an alternative test of the particular property of the product and the requirement made of the particular property, a second classification is determined. Depending on the first classification and the second classification, an accuracy or robustness of the simulation is determined.