Abnormal Log Event Clustering for Automated Failure Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing IT systems face challenges in predicting system failures, which often rely on manual expert review and traditional machine learning methods that are labor-intensive and difficult to adapt across different IT fields.
Innovation Solution
A method utilizing a sentence2vec model and K-Nearest Neighbor algorithm to determine current event clusters from log lines, followed by a Markov Chain model to predict the time of transition to abnormal event clusters, enabling automated detection and prediction of abnormal log events.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual expert review and traditional machine learning methods are used for predicting system failures, then prediction capability is achieved, but labor intensity and difficulty in adaptation increase
Solution Approach 1:
The system performs automated event cluster determination and anomaly detection without requiring manual expert intervention. The sentence2vec model automatically processes log lines to determine event clusters, and the system self-manages the entire prediction workflow, eliminating the need for continuous manual expert review while maintaining prediction capability
Solution Approach 2:
The sentence2vec model and event clustering approach provide a universal framework that can be applied across different IT fields and system types. The method processes various log formats and event types through a unified approach, making the system adaptable to different domains without requiring field-specific customization
2Reliability
If traditional machine learning methods are used for system failure prediction, then prediction is achieved, but adaptability across different IT fields decreases
Solution Approach 1:
The sentence2vec model creates a universal representation framework that handles diverse log formats and event types across different IT fields. The event clustering mechanism works consistently across various system types, enabling the same methodology to be applied universally while maintaining prediction accuracy across different domains
3Measurement precision
If manual expert review is used for detecting abnormal log events, then detection accuracy is maintained, but productivity and resource efficiency decrease
Solution Approach 1:
The system automatically determines event clusters and detects anomalies without requiring manual expert review. The automated pipeline processes log lines through sentence2vec encoding, event cluster determination, and anomaly detection continuously and efficiently, dramatically improving productivity while maintaining detection accuracy through sophisticated algorithmic analysis
Solution Approach 2:
The patent replaces manual expert review (mechanical human process) with automated computational methods including sentence2vec modeling, event clustering algorithms, and machine learning-based anomaly detection. This substitution maintains or improves detection accuracy while dramatically increasing processing speed and productivity
Data Source
AI summary
The embodiments of the present disclosure disclose a computer-implemented method, computer system and a computer program product for detecting and predicting an abnormal log event. In the method, a current event cluster from a plurality of event clusters for a log line in a log file is determined. The plurality of event clusters include at least one abnormal event cluster. Then, a time of event transition from the current event cluster to at least one abnormal event cluster is predicted.