Estimation Model Monitoring with Synthetic Test Data Alerts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data-driven machine learning models for critical systems face challenges in verifying robustness and stability, especially outside their predefined operational domains, due to limited formal methods and the impracticality of multiplying dataset sizes for verification.
Innovation Solution
A predictive monitoring device that generates test data by applying transformations to acquired data, such as adding noise or generating abnormal data, to evaluate the performance of estimation models, and issues alerts based on comparison results, ensuring robustness and stability within the predefined operational domain.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data-driven machine learning models are used for critical systems, then estimation accuracy is improved, but verification of robustness and stability becomes difficult
Solution Approach 1:
The patent applies preliminary action by generating test data with abnormal values before actual operation to verify robustness, and by establishing monitoring rules in advance that automatically trigger warnings when abnormal patterns are detected during operation. This proactive approach enables verification of machine learning model robustness without requiring extensive post-deployment testing.
Solution Approach 2:
The patent creates a virtual copy of the operational environment by generating synthetic test data that replicates real-world conditions including abnormal scenarios. This copying approach allows comprehensive verification of model robustness across various edge cases without needing to physically test every possible abnormal condition in critical systems.
2Reliability
If the number of data in learning datasets is increased to improve verification, then robustness verification is improved, but industrial cost and data availability become limiting factors
Solution Approach 1:
The patent generates synthetic test data that copies and replicates real operational data patterns, including normal and abnormal conditions. This synthetic data generation approach provides unlimited verification scenarios without requiring proportional increases in real data collection resources, effectively decoupling verification quality from data collection complexity.
Solution Approach 2:
The patent systematically varies parameters in generated test data to create diverse abnormal scenarios for robustness verification. By changing data parameters such as sensor values, environmental conditions, and operational states, the system achieves comprehensive verification coverage without needing to collect exponentially more real-world data.
3Adaptability or versatility
If machine learning models operate outside predefined operational domains, then adaptability is improved, but robustness becomes difficult to verify
Solution Approach 1:
The patent establishes monitoring rules and warning thresholds before the model operates outside its predefined operational domain. By preparing detection mechanisms in advance for boundary conditions and abnormal patterns, the system can verify robustness even when operating in extended domains, maintaining reliability checks alongside adaptability.
Solution Approach 2:
The patent implements continuous feedback through monitoring rules that detect when input data approaches or exceeds operational domain boundaries. This feedback mechanism triggers warnings and can initiate corrective actions, allowing the system to maintain verified robustness while adapting to operate in extended domains through active monitoring and control.
Data Source
AI summary
A predictive monitoring device for operation of a unit for estimating a physical quantity receiving data acquired by a detection unit and applying an estimation model to the acquired data, the predictive monitoring device including a generation unit for test data, applying a transformation to the acquired data, a computation unit estimating a value of the physical quantity by applying the model to the test data, an analysis unit for the test value, performing a comparison between the test value and an expected value according to a criterion for evaluating the performance of the model, and a warning unit issuing an alert according to the result of the comparison.