System Log Parser Generation Using Clustering and Regex Patterns
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Parsing unstructured and semi-structured system log messages is a manual and time-consuming process, and existing pattern recognition algorithms often fail to accurately handle variations in similar log messages, leading to incorrect patterns and inefficient parsing scripts.
Innovation Solution
An event management center utilizes clustering models to identify patterns in system log messages, forming clusters of similar types, removing variable attributes, and generating regex patterns for automated parsing scripts and rules, which can be applied to future messages without manual intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual parsing script generation is used, then parsing accuracy can be maintained, but time consumption and labor intensity increase significantly
Solution Approach 1:
The system enables self-service automated generation of parsing scripts through clustering algorithms that automatically analyze log messages, identify patterns, and generate regex-based parsing rules without requiring manual intervention for each script creation
Solution Approach 2:
The patent replaces the manual mechanical process of parsing script creation with an automated computational system using clustering algorithms and machine learning models that process log messages and generate parsing rules automatically
2Quantity of substance
If typical pattern recognition algorithms are used, then a substantial sample size can be processed, but accuracy decreases for slightly varying system log messages
Solution Approach 1:
The system changes the approach from requiring large sample sizes to working effectively with small samples by using clustering algorithms that identify structural patterns and variable attributes, allowing accurate pattern recognition even with limited log message data
Solution Approach 2:
The patent segments log messages into common features and variable attributes, allowing the system to recognize patterns in the structured portions while accommodating variations in the variable portions, thereby improving accuracy for slightly varying messages
3Adaptability or versatility
If parsing scripts are generated for each computing device, then device-specific log formats can be handled, but the process becomes extremely time-consuming
Solution Approach 1:
The system creates a universal parsing framework that can handle multiple device-specific log formats through a single automated platform. The clustering algorithms identify common patterns across different devices while accommodating device-specific variations, eliminating the need for separate manual script generation for each device
Data Source
AI summary
The present disclosure provides systems and methods for generation of parsing scripts or rules for unstructured or semi-structured system log messages, including systems and methods for identifying and clustering of same or substantially similar system log messages using machine learning. Patterns indicative of the same or substantially similar types system log messages can be generated based on the clustering of the system log messages and calculated similarities of attributes or distances between common features/fields of the system log messages, with the results of the clustering presented for analysis and development or adjustment of parsing scripts.