Universal Log Parser Using Message Signatures and Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current log parsing technologies require significant human effort and are limited by the need for source-specific plug-ins, making it costly and inefficient to extract meaningful information from varied log formats across different appliances.
Innovation Solution
A parser is trained to categorize and generalize log messages, generating message signatures that allow for parsing of arbitrary log sources without the need for source-specific plug-ins, using techniques like message tokenization, clustering, hashing, and probabilistic matching to extract data types and structures automatically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If source-specific plug-ins are created for each log format, then parsing accuracy is improved, but device complexity and maintenance cost increase
Solution Approach 1:
The patent implements a universal log parsing system that can handle multiple log formats without requiring source-specific plug-ins. The system uses machine learning models trained on diverse log data to generalize parsing capabilities across different appliances and log formats, eliminating the need for separate plug-ins for each log source while maintaining high parsing accuracy.
Solution Approach 2:
The system automatically adapts to new log formats through self-training mechanisms. When encountering unfamiliar log sources, the parser can autonomously learn the log format patterns and adjust its parsing rules without requiring manual intervention or development of new plug-ins, thereby reducing system complexity and maintenance burden.
2Measurement precision
If source-specific plug-ins are created for each log format, then parsing accuracy is improved, but maintenance cost and time increase
Solution Approach 1:
The system automatically adapts to new log formats through self-training mechanisms. When encountering unfamiliar log sources, the parser can autonomously learn the log format patterns and adjust its parsing rules without requiring manual intervention or development of new plug-ins, thereby reducing system complexity and maintenance burden.
Solution Approach 2:
The system performs preliminary training on diverse log formats during the initialization phase, pre-learning common patterns and structures across different appliances. This preliminary action enables the parser to handle a wide range of log formats out-of-the-box, reducing the need for future maintenance and adaptation work.
3Adaptability or versatility
If a universal parser without plug-ins is implemented, then ease of operation and adaptability improve, but parsing precision may deteriorate
Solution Approach 1:
The parsing system dynamically adapts its behavior based on the input log format. It uses machine learning models that can adjust their parameters and structures according to the specific characteristics of each log source, enabling the universal parser to achieve high accuracy for different log formats without requiring plug-in modifications.
Solution Approach 2:
The system changes its internal parameters and parsing rules based on the detected log format. By automatically identifying log source characteristics and adjusting parsing parameters accordingly, the universal parser maintains high precision across diverse log formats without needing source-specific configurations.
4Reliability
If manual plug-in development is required for each log source, then parsing reliability is improved, but productivity decreases
Solution Approach 1:
The system automatically adapts to new log formats through self-training mechanisms. When encountering unfamiliar log sources, the parser can autonomously learn the log format patterns and adjust its parsing rules without requiring manual intervention or development of new plug-ins, thereby reducing system complexity and maintenance burden.
Solution Approach 2:
The system performs preliminary training on diverse log formats during the initialization phase, pre-learning common patterns and structures across different appliances. This preliminary action enables the parser to handle a wide range of log formats out-of-the-box, reducing the need for future maintenance and adaptation work.
Data Source
AI summary
Herein are techniques for training a parser by categorizing and generalizing messages and abstracting message templates for parsing after training. In an embodiment, a computer generates a message signature based on a message sequence of tokens that were extracted from a training message. The message signature is matched to a cluster signature that represents messages of one of many clusters that have distinct signatures. The training message is added to the cluster. Based on a data type of the cluster signature, a value is extracted from a second message, such as a live message after training. Fuzzy signatures may be probabilistically matched to select a best matching cluster for a message. The value range of a token may be broadened or narrowed by adding or removing candidate data types, by adding or removing literals to a data type, and/or by promoting a narrow data type to a broader data type.