Automatic Online Log Template Mining with Wildcards
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log analysis solutions struggle with scalability and handling changes in token position and message length, failing to provide effective online log message aggregation in enterprise systems.
Innovation Solution
A method and system using token-based edit distance and density-based online clustering to determine similarity scores for incoming log messages, replacing tokens with wildcards for imperfect matches, and creating new templates for non-matching messages, enabling efficient and scalable log aggregation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing log analysis solutions use token-based matching to identify message templates, then message aggregation is achieved, but the system fails to handle changes in token position and message length
Solution Approach 1:
The patent transforms the rigid token matching approach into a flexible edit distance-based similarity scoring system. By calculating the number of edit operations (insertions, deletions, substitutions) needed to transform one message into another, the system adapts to variations in token position and message length while maintaining reliable matching through threshold-based classification of similarity scores.
Solution Approach 2:
The system dynamically adjusts template representations by replacing specific tokens with wildcards when imperfect matches are identified. This dynamic transformation allows the template to become more general and accommodate future variations, resolving the contradiction between adaptability and reliability by making the matching criteria flexible rather than fixed.
2Productivity
If existing log analysis solutions process log messages sequentially, then detailed analysis is possible, but the system cannot keep up with high volumes of log message data
Solution Approach 1:
The patent segments the log message processing into independent parallel operations. Multiple log messages are processed simultaneously through parallel execution of similarity score determination and template matching operations, eliminating sequential bottlenecks and enabling the system to handle high volumes of log data without sacrificing processing quality.
3Measurement precision
If existing log analysis solutions use complex matching algorithms to improve accuracy, then better template identification is achieved, but scalability is poor
Solution Approach 1:
The patent replaces complex mechanical-style string matching algorithms with a mathematical edit distance approach. By using standardized edit distance calculations and similarity score thresholds, the system achieves accurate template identification with computationally efficient operations that scale well, substituting algorithmic complexity with mathematical simplicity.
Data Source
AI summary
Methods and systems for log message aggregation include determining a first similarity distance score for a first incoming message by comparing the first incoming message to one or more stored templates. It is determined that the first incoming message imperfectly matches a matched template of the one or more stored templates, based on the first similarity distance score. A token in the imperfectly matched template is replaced with a wildcard, to reduce the first similarity distance score.