Unsupervised Log Clustering for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for detecting anomalies in computer systems based on log messages are inefficient due to manual configuration requirements, high costs, and instability in results, especially when scaling or adapting to changing systems and environments.
Innovation Solution
A system that automatically clusters log messages using the tri-point clustering technique, forming sequences of events and converting them into associations feature vectors to detect anomalies with minimal user involvement, and triggers alerts for anomalies such as hardware failures, software failures, or malicious activities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual configuration of log message parsers is used, then detection precision is improved, but device complexity and time consumption increase significantly
Solution Approach 1:
The system performs self-service by automatically generating parsers through unsupervised clustering of log messages without requiring manual configuration. The clustering algorithm autonomously identifies patterns and creates detection rules, eliminating the need for expert intervention while maintaining high detection precision.
Solution Approach 2:
The patent replaces the mechanical manual configuration process with an automated computational system. Instead of manually creating and configuring parsers, the system uses unsupervised learning algorithms to automatically generate parsers from raw log messages, substituting human effort with machine intelligence.
2Measurement precision
If manually labeled training data is used for classification, then detection precision is improved, but loss of time and cost increase
Solution Approach 1:
The patent inverts the traditional supervised learning approach by using unsupervised clustering instead. Rather than starting with labeled data and training a classifier, the system starts with unlabeled log messages and automatically discovers event patterns through clustering, eliminating the time-consuming labeling process while maintaining classification accuracy.
3Productivity
If traditional clustering techniques are used, then log messages are grouped into events, but result stability deteriorates due to parameter sensitivity
Solution Approach 1:
The system incorporates feedback mechanisms where clustering results are continuously evaluated and refined. The unsupervised clustering algorithm iteratively adjusts cluster assignments based on similarity metrics, ensuring stable and consistent event formation across different runs while maintaining high processing efficiency.
4Speed
If bag-of-words models are used for clustering, then processing speed is improved, but measurement precision deteriorates due to ignoring message order
Solution Approach 1:
The patent segments log messages into meaningful components while preserving their sequential relationships. Instead of treating messages as unordered word bags, the system maintains the temporal and contextual structure of log sequences, enabling accurate similarity measurement that respects message order and improves detection precision.
Data Source
AI summary
The disclosed embodiments provide a system that detects an anomaly in a computer system based on log messages. During operation, the system receives log messages generated by the computer system during operation of the computer system. Next, the system maps each received log message to a cluster in a set of clusters of log messages, wherein each cluster is associated with a specific event. The system then forms events for consecutive log messages into sequences of events. Finally, the system performs anomaly detection based on the sequences of events, wherein if an anomaly is detected, the system triggers an alert.