Statistical Model for Heterogeneous Log Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for analyzing log messages, particularly in IoT systems, face challenges in detecting anomalies due to the unstructured nature of logs and the lack of a standard logging mechanism, leading to high false positive results and inefficiencies in troubleshooting.
Innovation Solution
A computer-implemented method and system that builds a statistical model based on training logs to detect content-level anomalies in testing logs, using a content model builder that parses logs into fields, assigns attributes, and leverages user feedback to improve accuracy, allowing for unsupervised and semi-supervised anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If every log is parsed into tokens based on delimiters and an index is maintained to identify unique tokens, then anomaly detection can be performed, but false positive results increase significantly
Solution Approach 1:
The patent segments log analysis into two levels: token-level segmentation (breaking logs into tokens) and sentence-level segmentation (grouping tokens into sentences). This hierarchical segmentation allows the system to capture both fine-grained token patterns and contextual sentence structures, improving anomaly detection while reducing false positives through multi-level verification
Solution Approach 2:
The patent introduces a new dimension by building statistical models not just on individual tokens but on sentence-level features including token frequency, position, and contextual relationships. This dimensional expansion from token-only analysis to sentence-level statistical analysis enables more accurate anomaly detection with fewer false positives
2Measurement precision
If a statistical model is built on training logs to detect content-level anomalies, then detection accuracy improves, but system complexity increases
Solution Approach 1:
The patent implements preliminary action by pre-processing training logs to build statistical models that capture normal behavior patterns before actual anomaly detection begins. The system pre-computes token frequencies, sentence structures, and contextual relationships during a training phase, so that during operation, anomaly detection can proceed efficiently without real-time complex computations
Solution Approach 2:
The patent introduces statistical models as intermediary components between raw log data and anomaly detection results. These models act as mediators that translate unstructured log content into structured statistical representations, simplifying the detection process while maintaining high accuracy through learned patterns from training data
3Productivity
If log messages are analyzed without considering unstructured free-form text nature, then processing speed improves, but detection accuracy deteriorates
Solution Approach 1:
The patent replaces mechanical text processing approaches with statistical modeling techniques. Instead of using rigid rule-based parsing and pattern matching, the system employs statistical models that learn from training data, automatically adapting to the unstructured nature of log messages. This substitution maintains processing speed while significantly improving detection accuracy through data-driven pattern recognition
Data Source
AI summary
A computer-implemented method for automatically analyzing log contents received via a network and detecting content-level anomalies is presented. The computer-implemented method includes building a statistical model based on contents of a set of training logs and detecting, based on the set of training logs, content-level anomalies for a set of testing logs. The method further includes maintaining an index and metadata, generating attributes for fields, editing model capability to incorporate user domain knowledge, detecting anomalies using field attributes, and improving anomaly quality by using user feedback.