Log Analysis System Structural Pattern Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log analysis systems struggle to accurately detect and analyze unknown failures in information processing systems, as they rely on known failure patterns and cannot distinguish between normal and abnormal operations effectively, leading to false incidents and incomplete failure analysis.
Innovation Solution
A log analysis system that compares structural patterns of log messages with known patterns, associates reference information, and generates new patterns from log message groups with different structural patterns, enabling accurate failure detection and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If log analysis systems rely on known failure patterns for detection, then detection accuracy for known failures is improved, but the ability to detect unknown failures deteriorates
Solution Approach 1:
The system performs preliminary actions by generating multiple candidate patterns from normal operation logs before actual failure detection is needed. These candidate patterns represent various normal operational sequences and are stored in advance, enabling the system to compare against unknown failure scenarios without requiring pre-defined failure patterns.
Solution Approach 2:
The system introduces an intermediary approach by using normal operation patterns as a mediator between known and unknown failures. Instead of directly comparing against known failure patterns, the system uses normal patterns to identify deviations, allowing detection of unknown failures through abnormal deviations from normal behavior rather than matching against predefined failure signatures.
2Productivity
If the system analyzes individual log messages separately, then processing speed is improved, but the accuracy of failure analysis deteriorates
Solution Approach 1:
The system segments the log analysis process into two distinct phases: first, it rapidly processes individual log messages to extract basic information; second, it groups related messages into sequences for comprehensive pattern matching. This segmentation allows the system to maintain high processing speed for individual messages while achieving high accuracy through sequence-level analysis.
Solution Approach 2:
The system transitions from one-dimensional individual message analysis to two-dimensional sequence analysis. By organizing log messages into temporal sequences and analyzing them as structured patterns rather than isolated units, the system adds a temporal and contextual dimension to the analysis, thereby improving accuracy without sacrificing the efficiency of individual message processing.
3Stability of the object's composition
If the system uses predefined analysis rules, then analysis consistency is improved, but the ability to adapt to new failure types deteriorates
Solution Approach 1:
The system implements self-service by automatically generating candidate patterns from normal operation logs without requiring manual rule creation. The system serves itself by learning normal operational patterns and using these to detect anomalies, enabling automatic adaptation to new failure types while maintaining consistency through systematic pattern generation rather than arbitrary rule-setting.
Solution Approach 2:
The system changes the fundamental parameter from fixed predefined rules to dynamic candidate patterns generated from actual system operations. By transforming the analysis basis from static rules to adaptive patterns that reflect real system behavior, the system maintains consistency through methodological rigor while gaining adaptability to new failure scenarios.
Data Source
AI summary
In order to specify and analyze a failure occurring in an information processing system with a high degree of accuracy, an log analysis system includes an analysis unit comparing a structural pattern of a log message group constituted by at least one log message with a known pattern, and associating, with the structural pattern of the log message group, reference information including information about the known pattern matching a structural pattern of another log message group supplementary the log message group, and a pattern generation unit generating a new pattern from the log message group having a structural pattern different from the known pattern in accordance with an analysis result given by the analysis unit.