Log Anomaly Detection Using Real-Time Statistical Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log anomaly detection methods are not suitable for large-scale IT systems with sophisticated incidents, as they have low detection accuracy, high false positive rates, and require extensive training data and time-consuming template learning processes, making them inefficient for real-time anomaly detection.
Innovation Solution
A real-time statistical model-based log anomaly detection method that uses fast entity extraction and word embeddings to create feature count vectors, allowing for immediate anomaly prediction and continuous model updating, reducing the need for large training datasets and avoiding biased models by skipping contaminated data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional rule-based log anomaly detection methods are used, then the system is simple to implement, but the detection accuracy is low and false positive rate is high
Solution Approach 1:
The patent replaces traditional rule-based mechanical detection systems with machine learning models that automatically learn patterns from log data. The system uses trained models to detect anomalies without relying on manually crafted rules, thereby improving detection accuracy while reducing false positives. The complexity is managed through automated model training and deployment pipelines.
Solution Approach 2:
The patent transforms the detection approach by changing from fixed rule parameters to dynamic statistical parameters learned from data. The system computes statistical distributions from historical logs and uses these learned parameters for anomaly detection, allowing the system to adapt to changing log patterns while maintaining high accuracy.
2Measurement precision
If extensive training data is collected for model learning, then the detection accuracy improves, but the training time and resource requirements increase significantly
Solution Approach 1:
The patent performs preliminary actions by pre-processing and storing statistical distributions from historical log data before actual anomaly detection is needed. The system pre-computes baseline statistics and stores them for rapid comparison during runtime, eliminating the need for time-consuming training during operational phases.
Solution Approach 2:
The patent uses partial action by selecting only the most relevant statistical features and log patterns for modeling, rather than processing all possible data. The system focuses on key entities and patterns that contribute most to anomaly detection, reducing training data requirements while maintaining high detection accuracy.
3Adaptability or versatility
If the statistical model is updated continuously to adapt to changing log patterns, then the adaptability improves, but the computational overhead and resource consumption increase
Solution Approach 1:
The patent implements periodic action by updating statistical models at scheduled intervals rather than continuously. The system re-trains models periodically using accumulated log data, balancing adaptability to changing patterns with reduced computational overhead compared to continuous updating.
Solution Approach 2:
The patent maintains continuity of useful action by continuously monitoring log data and computing incremental statistical updates without full model re-training. The system continuously refines anomaly detection by accumulating statistics and making incremental adjustments, ensuring adaptability while minimizing resource consumption.
4Adaptability or versatility
If manual rule-based anomaly detection is used, then the system requires minimal computational resources, but it cannot handle sophisticated system incidents in large-scale IT systems
Solution Approach 1:
The patent replaces manual rule-based detection with automated machine learning systems that can handle sophisticated incidents. The system uses trained models to automatically analyze complex log patterns and detect anomalies that would be difficult to capture with manual rules, significantly improving the capability to handle sophisticated system incidents.
Solution Approach 2:
The patent implements self-service by enabling the system to automatically learn from log data and improve its detection capabilities without manual intervention. The system autonomously trains models, updates statistical distributions, and adapts to new patterns, reducing the need for manual rule creation and maintenance while improving detection efficiency.
Data Source
AI summary
A computer-implemented method, a computer program product, and a computer system for log anomaly detection. A computer receives a windowed log of incoming raw log messages. A computer compares statistical distribution metrics of entities in the windowed log with a statistical distribution extracted from a real-time statistical model for the entities. In response to the statistical distribution metrics being statistically different from the statistical distribution extracted from the real-time statistical model for the entities, a computer tags the windowed log as an entity anomaly. A computer computes a distance between an average word embedding vector in the windowed log and a statistical distribution extracted form a real-time statistical model for word embeddings. In response to the distance being greater than a predetermined threshold, a computer tags the windowed log as a word embedding anomaly. A computer sends to a user an alert with an anomaly severity level.