Time-Series Anomaly Detection for Rare Failure Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large scale enterprise computing systems face complex failure conditions that are difficult to determine, requiring significant time and resources for triage and root cause analysis, exacerbated by the variety of factors leading to failures and the computational infeasibility of running multiple machine learning models simultaneously, and the low accuracy of existing ML models in detecting rare failures.
Innovation Solution
A dual-layered approach using an anomaly detection engine and a supervised classification suite, where the anomaly detection component identifies anomalous data points using unsupervised ML models, and the event prediction component predicts failure events using supervised ML models, reducing computational complexity and improving accuracy by running models on demand based on detected anomalies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple machine learning models are run simultaneously to improve failure detection accuracy, then detection precision is improved, but computational resources and time consumption increase significantly
Solution Approach 1:
The system segments the failure detection process into two distinct stages: anomaly detection (identifying unusual data points) and failure prediction (determining likelihood of future failures). Each stage uses specialized machine learning models trained for specific purposes, allowing the system to achieve high detection accuracy while managing computational complexity through staged processing rather than running all models simultaneously on all data.
2Measurement precision
If comprehensive analysis of all failure factors is performed to improve root cause analysis accuracy, then measurement precision is improved, but time consumption increases
Solution Approach 1:
The system performs preliminary anomaly detection on incoming data streams to identify unusual patterns before conducting comprehensive failure analysis. This preliminary filtering action allows the system to focus subsequent detailed analysis only on data points that exhibit anomalous characteristics, significantly reducing the time required for root cause analysis while maintaining comprehensive coverage of potential failure factors.
3Ease of manufacture
If existing machine learning models are used for rare failure detection, then implementation simplicity is maintained, but detection accuracy deteriorates due to low accuracy in detecting rare events
Solution Approach 1:
The system changes the approach to detecting rare failures by using anomaly detection models that identify unusual data patterns regardless of how rare they are. Instead of relying on models trained to detect specific rare failure types (which require extensive labeled training data), the system uses unsupervised anomaly detection to flag unusual behavior, then applies failure prediction models only to these anomalous cases, dramatically improving rare failure detection accuracy while maintaining implementation feasibility.
Data Source
AI summary
A method for time series anomaly detection with rare event failure prediction includes determining, by a device including a processor and using a first machine learning model, a location of a data anomaly within sequential data. The method also includes determining, by the device and using a second machine learning model that is not the first machine learning model, whether a probability of a future failure event is at least a threshold probability based on the data anomaly and the sequential data.