Log Message Parsing via Format Clustering and Rule Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Classifying log messages with different formats in a log message stream is challenging due to the difficulty in identifying and parsing these messages, which hinders effective debugging and system behavior understanding in computer systems.
Innovation Solution
A multi-format log message parsing system that includes a rule identifier to detect delimiters, a log message portion selector to cluster similar messages, a substring partitioner to identify substrings, and a parsing rule generator to create regular expressions for each cluster, enabling efficient classification and parsing of log messages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If log messages with different formats are received in a log message stream, then the system can collect diverse log data from multiple source components, but the difficulty in identifying and parsing these messages increases
Solution Approach 1:
The patent segments log messages into clusters based on format similarity, creating groups of messages with identical or similar structures. This segmentation allows the system to handle diverse log formats by processing each cluster separately with format-specific parsing rules, thereby reducing the overall difficulty of parsing while maintaining adaptability to multiple formats.
Solution Approach 2:
The patent introduces an intermediary classification mechanism that acts as a mediator between raw log messages and parsing operations. The system classifies messages into format-based clusters before parsing, and this classification layer serves as an intermediary that simplifies the parsing task by pre-organizing messages according to their formats.
2Measurement precision
If multiple parsing rules are generated for different log message formats, then the classification accuracy improves, but the system complexity increases
Solution Approach 1:
The patent segments the parsing system into multiple format-specific clusters, each with its own parsing rules. This segmentation allows high classification accuracy for each cluster while managing overall system complexity by organizing parsing logic into modular, format-specific components rather than a single complex parser.
Solution Approach 2:
The patent performs preliminary classification of log messages into format-based clusters before applying parsing rules. This preliminary action organizes messages by format, allowing simpler parsing operations to be applied to each cluster, thereby achieving high classification accuracy while reducing the complexity of individual parsing operations.
3Productivity
If log messages are clustered by format, then the parsing efficiency improves, but the processing time for initial classification increases
Solution Approach 1:
The patent performs preliminary classification of log messages into format-based clusters before parsing. While this initial classification requires processing time, it enables significantly more efficient parsing operations afterward by organizing messages according to their formats, allowing the system to quickly apply appropriate parsing rules to each cluster.
Solution Approach 2:
The patent segments log processing into distinct phases: classification into format clusters, followed by format-specific parsing. This segmentation allows the system to invest processing time in the initial classification phase, which then enables highly efficient parsing operations in subsequent phases, improving overall productivity.
Data Source
AI summary
A plurality of log messages may be clustered into a plurality of clusters. For each of the plurality of log messages, the log message may be partitioned into a series of substrings. At least two of the plurality of clusters may be selected. For each one of the at least two selected clusters, a parsing rule may be generated corresponding to a plurality of substrings each of which are at a given location of a respective one of the log messages of the plurality of log messages in the one of the selected cluster.