Signal Metric Selection for Simulation Model Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The reliability of simulation models in model-based testing is limited due to a lack of confidence in their predictive quality, and it is unclear which signal metrics to use for validating these models, especially when performance indicators for the system under test are not established.
Innovation Solution
A method is developed to select the most suitable signal metric based on a specific Key Performance Index (KPI) by calculating the interrelation between varying performance indices and signal metrics, using a modified correlation coefficient to handle cases of constant variance, and allowing for automatic error correction in simulation models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If simulation models are used for testing in model-based testing, then testing activities can be automated and test artifacts can be generated, but the reliability of the simulation models is limited due to a lack of confidence in their predictive quality
Solution Approach 1:
The patent applies preliminary action by performing validation of simulation models before they are used for testing. The method calculates performance indices from simulation results and compares them with measured values from real systems in advance, establishing confidence in the models' predictive quality before deployment. This preliminary validation ensures that only reliable simulation models are used for automated testing, thereby resolving the contradiction between automation productivity and model reliability.
2Measurement precision
If multiple signal metrics are used to validate simulation models, then the evaluation can be comprehensive, but it is unclear which signal metrics to use when performance indicators for the system under test are not established
Solution Approach 1:
The patent applies parameter changes by dynamically selecting signal metrics based on the specific performance indicators being validated. The method adapts the choice of signal metrics (such as magnitude comparison, phase shift, or correlation metrics) according to the characteristics of the performance index and the type of validation required. This flexible parameter selection approach enables comprehensive validation while avoiding the complexity of evaluating all possible metrics, as only the most relevant metrics are applied in each case.
3Measurement precision
If signal metrics are used to compare simulation results with real measurements, then the similarity between signals can be quantified, but the choice of metric can be trivial or unclear when threshold values are not established
Solution Approach 1:
The patent applies feedback by using the calculated performance indices to guide the selection and application of appropriate signal metrics. The method establishes a feedback loop where simulation results are validated against real measurements, and the outcomes inform subsequent validation efforts. This feedback mechanism helps establish meaningful threshold values based on actual performance data, making metric selection more intuitive and operationally clear while maintaining precise quantification of signal similarity.
Data Source
AI summary
A method for evaluating a simulation model. In the method, for selected test cases, a first performance index is calculated in the simulation model. For the same test case, a second performance index is ascertained in a real test environment. For each of the test cases, a difference is calculated between the first performance index and the second performance index, and a signal metric is determined. For each of the signal metrics, an interrelation between the difference and the respective signal metric is investigated. The signal metric that exhibits the closest interrelation with the difference is selected.