Failure Prediction in Distributed Systems Using ML Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Failure detection in distributed systems is complex and typically reactive, relying on assumptions that not all components are accessible, limiting proactive failure prediction and remediation.
Innovation Solution
A method using a failure detection platform that obtains time series data, identifies anomalies, trains machine-learning models to classify network failures, and predicts future system failures, enabling proactive remedial actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional reactive failure detection is used in distributed systems, then system complexity is reduced, but failure prediction accuracy deteriorates
Solution Approach 1:
The patent segments the failure detection system into multiple independent machine learning models, each specializing in different failure patterns and data types. This segmentation allows complex analysis to be distributed across multiple simpler models, maintaining low overall system complexity while achieving high prediction accuracy through coordinated model outputs.
Solution Approach 2:
The patent introduces an intermediary layer of anomaly detection that processes raw time series data before feeding it to classification models. This intermediary anomaly detection mechanism simplifies the input for subsequent models while improving prediction accuracy by filtering and preprocessing data to highlight relevant failure indicators.
2Measurement precision
If comprehensive component access is assumed for failure detection, then failure detection accuracy is improved, but system adaptability deteriorates
Solution Approach 1:
The patent creates a universal failure detection platform that can adapt to multiple distributed system configurations and component types. The machine learning models are designed to process various data formats and system architectures, allowing the system to maintain high detection accuracy across diverse environments without requiring access to every specific component.
Solution Approach 2:
The patent employs parameter changes by dynamically adjusting model inputs, thresholds, and configuration based on the specific distributed system being monitored. This allows the system to adapt to different architectures and access limitations while maintaining accurate failure detection through flexible parameter adjustment rather than rigid component access requirements.
3Measurement precision
If multiple machine learning models are trained for failure prediction, then prediction accuracy is improved, but training time increases
Solution Approach 1:
The patent applies preliminary action by pre-training base models on historical data and storing them for rapid deployment. When new failure patterns emerge, the system fine-tunes existing models rather than training from scratch, significantly reducing training time while maintaining improved prediction accuracy through selective model updates.
Solution Approach 2:
The patent uses partial action by training and deploying only the specific subset of models needed for particular failure scenarios. Rather than training all possible models continuously, the system activates only relevant models based on detected anomaly types, reducing overall training time while maintaining high accuracy for each specific failure mode.
Data Source
AI summary
Embodiments of the invention are directed to systems, method, and devices for detecting failures in distributed systems. A failure detection platform may identify anomalies in time series data, the time series data corresponding to historical network messages. The anomalies can be labeled and used to train a first predictive model. At least one other model may be trained using the time series data, the anomaly labels and a supervised machine-learning algorithm. A third model can be trained to identify a system failure based at least in part on the outputs provided by the first and the second model. The third model, once trained, can be utilized to predict a future system failure.