Log Pattern Comparative Analysis Engine for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log-based system maintenance technologies face challenges in extracting actionable insights from heterogeneous operation logs due to the redundancy of semantically similar but syntactically different log patterns, which hinders effective anomaly detection and system failure classification.
Innovation Solution
A content-aware heterogeneous log pattern comparative analysis method that extracts log syntactic patterns, generates latent representation vectors, and predicts anomalies using clustering techniques, allowing for the identification and removal of semantically similar patterns to improve anomaly detection and system failure classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If log patterns are extracted from heterogeneous logs without semantic analysis, then extraction speed is improved, but accuracy of anomaly detection deteriorates due to redundancy of semantically similar patterns
Solution Approach 1:
The patent replaces traditional mechanical text processing methods with deep learning-based semantic analysis. Specifically, it uses pre-trained language models (BERT, RoBERTa) to automatically understand and compare semantic meanings of log patterns, substituting manual or rule-based semantic analysis with automated neural network-based semantic comprehension, thereby resolving the contradiction between extraction speed and detection accuracy
Solution Approach 2:
The patent transforms log patterns from raw text form into numerical embedding vectors through pre-trained language models. This parameter transformation enables quantitative comparison of semantic similarity between different log patterns, allowing the system to efficiently identify and remove redundant patterns while maintaining high anomaly detection accuracy
2Loss of information
If all heterogeneous log patterns are retained for analysis, then completeness of log information is improved, but system complexity increases due to redundant semantically similar patterns
Solution Approach 1:
The patent implements a log pattern deduplication mechanism that identifies and removes semantically redundant log patterns while preserving the essential information content. By comparing semantic embeddings of log patterns and filtering out duplicates, the system recovers from the initial complete but redundant log set to a streamlined set that maintains information completeness while reducing complexity
Solution Approach 2:
The patent creates a universal semantic embedding space that can represent diverse log patterns from multiple sources in a unified framework. This universal representation enables the system to handle heterogeneous logs from different subsystems using the same semantic analysis mechanism, reducing overall system complexity while maintaining comprehensive log coverage
3Ease of operation
If traditional log analysis methods are used, then ease of operation is improved, but productivity decreases due to inability to effectively classify system failures
Solution Approach 1:
The patent replaces traditional manual or rule-based log analysis methods with automated deep learning-based semantic analysis. The system automatically computes semantic embeddings, compares patterns, identifies anomalies, and classifies failures without requiring manual intervention, thereby maintaining ease of operation while dramatically improving failure classification productivity
Solution Approach 2:
The patent implements self-service automation where the system automatically performs semantic analysis, pattern comparison, anomaly detection, and failure classification without human intervention. The pre-trained language models and automated clustering algorithms enable the system to serve itself in analyzing logs and identifying issues, improving productivity while keeping the interface simple for users
Data Source
AI summary
A computer-implemented method, system, and computer program product are provided for content aware heterogeneous log pattern comparative analysis. The method includes receiving, by a processor-device, a plurality of heterogeneous logs. The method also includes extracting, by the processor-device, a plurality of log syntactic patterns from the plurality of heterogenous logs. The method additionally includes generating, by the processor-device, latent representation vectors for each of the plurality of log syntactic patterns. The method further includes predicting, by the processor-device, an anomaly from the clustered latent representation vectors. The method also includes controlling an operation of a processor-based machine to react in accordance with the anomaly.