Log Parsing Template Generation via Multi-Stage Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional log parsing methods rely on rigid templates and exact matching, leading to inaccuracies in identifying invariants and parameters, especially in complex microservice systems with large volumes of logs, resulting in reduced quality of log analysis outputs and user experience.
Innovation Solution
A method that performs a course analysis to identify invariants, parameters, and gray area terms, followed by a fine sequence labeling analysis to classify these terms and generate log parsing templates, and then combines similar templates using similarity analysis to improve clustering and parsing accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If rigid templates and exact matching are used for log parsing, then the parsing process is simple and fast, but the accuracy in identifying invariants and parameters deteriorates
Solution Approach 1:
The log parsing process is divided into multiple stages: course analysis to identify candidate invariants and parameters, fine analysis to precisely classify gray area terms, and template generation. This segmentation allows each stage to focus on specific tasks, improving overall accuracy without sacrificing too much speed.
Solution Approach 2:
The patent introduces an intermediary classification stage that categorizes terms into three groups: definite invariants, definite parameters, and gray area terms. This intermediary step acts as a mediator between rough matching and final classification, enabling more accurate identification while maintaining parsing efficiency.
2Measurement precision
If multiple analysis stages are performed to improve log parsing accuracy, then the identification precision improves, but the processing time increases
Solution Approach 1:
The patent applies partial action by performing different levels of analysis on different terms. Definite invariants and parameters are identified through course analysis alone, while only gray area terms require fine analysis. This selective approach improves accuracy for ambiguous cases without unnecessarily processing all terms, reducing overall processing time.
Solution Approach 2:
Different analysis depths are applied to different portions of the log data based on their characteristics. Terms with clear patterns receive minimal processing, while ambiguous terms receive more intensive fine analysis. This local quality approach optimizes the balance between accuracy and processing time for each specific term.
3Productivity
If traditional exact matching is used for log clustering, then the clustering process is computationally efficient, but the clustering quality and anomaly detection accuracy deteriorate
Solution Approach 1:
The patent changes the matching parameters from exact string matching to similarity-based matching with configurable thresholds. This allows the system to identify semantically similar log entries even with minor variations, improving clustering quality and anomaly detection while maintaining reasonable computational efficiency through optimized similarity calculations.
Data Source
AI summary
Aspects of the disclosure provide for a method. In at least some examples, the method includes receiving logs of a computer system, wherein a log comprises information regarding an operation of the computer system. The method also includes performing a course analysis of the logs to identify invariants, parameters, and gray area terms included in the logs, wherein gray area terms are terms included in the logs that are undefined between identifications of invariant or parameter by the course analysis. The method also includes performing a fine analysis to identify each of the gray area terms as either an invariant or a parameter to generate first log parsing templates, wherein the fine analysis is a sequence labeling analysis. The method also includes performing a similarity analysis to combine similar templates from among the first log parsing templates to form second log templates.