ML Model Dependency Checking Using Probe Signal Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning (ML) models used in prognostic surveillance for asset monitoring often suffer from inaccurate predictions due to dependency phenomena like 'following' and 'spillover', leading to excessive false or missed alarms, which can be catastrophic in critical systems.
Innovation Solution
A model dependency check system applies an oscillating perturbation to input signals and monitors the output for the presence of this perturbation, using cross-power spectral density (CPSD) analysis to detect and quantify dependencies, thereby evaluating the ML model's accuracy and recommending retraining or mitigation techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If ML models are used for prognostic surveillance, then detection capability is improved, but false alarm rate increases due to dependency phenomena
Solution Approach 1:
The patent applies preliminary action by performing dependency checking on ML models before deployment in prognostic surveillance. The system checks for dependency phenomena in the trained ML model and identifies problematic dependencies before they can cause false alarms, allowing corrective actions to be taken in advance.
Solution Approach 2:
The patent introduces an intermediary dependency checking system that acts as a mediator between the ML model and the prognostic surveillance application. This intermediary layer analyzes the ML model's behavior, identifies dependency phenomena, and provides feedback to mitigate false alarms without directly interfering with the core surveillance function.
2Reliability
If ML models are used for prognostic surveillance, then detection capability is improved, but missed alarm rate increases due to dependency phenomena
Solution Approach 1:
The dependency checking system performs preliminary analysis of the ML model to identify dependencies that could cause missed alarms before deployment. By detecting these issues in advance, the system can apply corrective measures to ensure reliable alarm detection.
Solution Approach 2:
The patent implements feedback mechanisms where the dependency checking system continuously monitors the ML model's performance and provides feedback on detected dependencies. This feedback loop enables ongoing mitigation of missed alarms through model retraining or parameter adjustment based on identified issues.
3Measurement precision
If dependency checking is performed on ML models, then prediction accuracy is improved, but system complexity increases
Solution Approach 1:
The dependency checking system is designed to be self-service, automatically evaluating ML models for dependency phenomena without requiring extensive manual intervention. The system autonomously performs the checking, analysis, and mitigation recommendations, reducing the complexity burden on users.
Solution Approach 2:
By introducing the dependency checking system as an intermediary layer, the patent isolates the complexity of model evaluation from the core prognostic surveillance system. This intermediary handles the complex dependency analysis separately, allowing the main system to remain relatively simple while still achieving high prediction accuracy.
Data Source
AI summary
Systems, methods, and other embodiments associated with associated with dependency checking for machine learning (ML) models are described. In one embodiment, a method includes applying a repeating probe signal to an input signal input into a machine learning model. An estimate signal output from the machine learning model is monitored, and the repeating probe signal is checked for in the estimate signal. Based on the results of the checking for the repeating probe signal, an evaluation of dependency in the machine learning model is presented.