System Log Parser Generation Using Message Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems require manual or partially automated processes to generate parsing scripts for unstructured and semi-structured system log messages, which are time-consuming and often result in inaccurate patterns due to the variety of similar yet slightly varying log messages.
Innovation Solution
An event management center utilizes clustering models to identify patterns in system log messages, forming clusters of similar types and generating regex patterns for automated parsing, with the option for user confirmation and editing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual or partially automated processes are used to generate parsing scripts, then flexibility in handling different log formats is maintained, but time consumption and labor effort increase significantly
Solution Approach 1:
The system enables self-service automated generation of parsing scripts by analyzing log messages and generating regex patterns without requiring manual intervention. The event management center automatically processes log messages, identifies patterns, and creates parsing scripts, eliminating the time-consuming manual process while maintaining adaptability to different log formats.
Solution Approach 2:
The patent replaces the mechanical manual process of creating parsing scripts with an automated computational system. The event management center uses algorithmic processing to analyze log messages and generate regex patterns, substituting human manual work with automated mechanical processing that is both faster and adaptable.
2Productivity
If typical pattern recognition algorithms are used, then processing speed is improved, but accuracy decreases due to inability to handle similar yet varying log messages
Solution Approach 1:
The system employs dynamic pattern recognition that can adapt to variations in log messages. Instead of using static rigid patterns, the event management center generates flexible regex patterns that can accommodate similar yet varying log messages from different computing devices and peripherals, maintaining both speed and accuracy.
Solution Approach 2:
The patent changes the parameters of pattern recognition by using regex patterns with variable components that can match different log message formats. The system adjusts pattern parameters to handle variations in log messages while maintaining processing efficiency, resolving the contradiction between speed and accuracy.
3Extent of automation
If clustering models are used to identify patterns in log messages, then automated parsing script generation is achieved, but system complexity increases
Solution Approach 1:
The event management center is segmented into distinct functional modules: log message reception, clustering model processing, pattern identification, and parsing script generation. This segmentation allows the complex automation process to be broken down into manageable components, reducing perceived complexity while maintaining high automation capability.
Data Source
AI summary
The present disclosure provides systems and methods for generation of parsing scripts or rules for unstructured or semi-structured system log messages, including systems and methods for identifying and clustering of same or substantially similar system log messages using machine learning. Patterns indicative of the same or substantially similar types system log messages can be generated based on the clustering of the system log messages and calculated similarities of attributes or distances between common features/fields of the system log messages, with the results of the clustering presented for analysis and development or adjustment of parsing scripts.