Log Template Parsing for Event Extraction Across Mixed Log Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log parsing methods fail to parse various original log data using the same rules and extract events triggered by the data, leading to incomplete analysis of variable changes due to differing parsing rule applications and content arrangements.
Innovation Solution
A log processing device and method that divides original log data into blocks, transforms numeric variables into representative codes based on data attributes, and combines continuous blocks to generate log templates corresponding to events, enabling consistent parsing and analysis of variable changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If specific symbols and preset parsing rules are used to replace variables in log data, then data compression is achieved, but the ability to parse various original log data using the same rules is lost
Solution Approach 1:
The log data is divided into multiple blocks according to a first rule, and each block is processed independently through segmentation. This allows the system to handle diverse log formats by treating each block as a separate unit that can be individually analyzed and compressed, thereby maintaining parsing consistency across different log types while achieving data compression.
Solution Approach 2:
The system uses a unified parsing mechanism that can handle various original log data formats through the same set of operations. By transforming numeric variables into representative codes based on data attributes and applying consistent block division rules, the system achieves universal parsing capability across different log types without requiring format-specific rules.
2Measurement precision
If different parsing rules are applied to original log data with different expressions, then parsing accuracy is improved, but device complexity increases
Solution Approach 1:
The system determines data integrity for each block based on its specific characteristics and applies appropriate combination operations locally. Each block is evaluated individually to determine whether it should be combined with adjacent blocks, allowing the parsing process to adapt to local data qualities while maintaining an overall simple and unified parsing framework.
3Productivity
If events are not extracted during the parsing process, then parsing speed is improved, but the ability to analyze variable changes is lost
Solution Approach 1:
The system performs preliminary actions during the parsing process by determining data integrity for each block and identifying events that triggered the original log data. This preliminary event extraction and integrity determination is done as part of the standard parsing flow, so no additional processing steps are needed later, thereby maintaining high parsing speed while preserving event information for subsequent analysis.
Data Source
AI summary
A log processing device and a log processing method thereof are provided. The log processing device divides the original log data into a plurality of block data, transforms a numeric variable of each of the block data into a representative code, and determines whether to perform a combination process for continuous block data to generate a plurality of combinational block data according to a data integrity of each of the block data. The log processing data takes the combinational block data as a log template, and each of the combinational block data corresponds to an event.