Automatic Log Parser Creation via Pattern Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The manual creation of parsers for normalizing log data from various IT sources is time-consuming and requires large teams, necessitating an automated method for parsing and normalizing event logs across different vendors and systems.
Innovation Solution
A system and method for automatically creating parsers by identifying log groups, tokenizing logs, grouping them based on patterns, and mapping normalized fields to regular expressions, with user interface support for vendor and event type association, and knowledgebase comparison for token identification and regular expression generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If parsers are created manually to normalize log data from various IT sources, then parsing accuracy and customization can be achieved, but the process becomes time-consuming and requires large teams
Solution Approach 1:
The system enables automatic parser creation by allowing the computer to autonomously analyze log formats, identify patterns, and generate normalization rules without human intervention. The automatic parser creation module processes log data from multiple sources and creates parsers automatically, eliminating the need for manual parser development while maintaining accuracy through pattern recognition and machine learning algorithms.
Solution Approach 2:
The system creates parsers by copying and adapting proven normalization patterns from existing parsers. When a new log format is encountered, the system compares it against known patterns in the parser library, copies relevant normalization rules, and adapts them to the new format, significantly reducing creation time while preserving parsing accuracy through pattern-based replication.
2Adaptability or versatility
If manual parser creation is used to handle diverse log formats from different vendors, then customization for specific log structures is possible, but the complexity and resource requirements increase significantly
Solution Approach 1:
The system implements a universal parser creation framework that can handle diverse log formats from multiple vendors through a single automated process. The automatic parser creation module analyzes various log structures (JSON, CSV, key-value pairs, structured text) and generates appropriate normalization rules automatically, eliminating the need for separate manual parser development processes for each log type and reducing overall system complexity.
Solution Approach 2:
The system adapts to different log formats by dynamically changing analysis parameters and normalization rules based on the detected log structure. The automatic parser creation module identifies log format characteristics and adjusts parsing parameters automatically, enabling versatile handling of diverse formats without increasing manual intervention complexity.
3Productivity
If large teams are employed to create parsers for multiple log sources, then comprehensive log normalization can be achieved, but operational costs and resource requirements increase
Solution Approach 1:
The system replaces the mechanical process of manual parser creation by human teams with an automated computational system. The automatic parser creation module performs log analysis, pattern recognition, and parser generation through computer algorithms, substituting human labor with automated processing that maintains high productivity while eliminating the need for large human teams dedicated to parser development.
Solution Approach 2:
The system enables self-service parser creation where the computer autonomously analyzes log data and generates normalization parsers without human intervention. This automated self-service capability maintains comprehensive log processing capacity while eliminating the need for large human teams, as the system independently handles parser creation for multiple log sources.
Data Source
AI summary
The present disclosure describes a system, method, and computer program for automatically creating a parser for a log group. A parser-creation system groups logs that do not satisfy conditions for an existing parser, enables a user to select a log group for parser creation, and automatically creates a parser for the selected log group. In creating a parser, the system extracts values and keys value pairs from the log group and identifies the corresponding normalized output fields and regular expressions for the values and key-value pairs. To identify normalized fields corresponding to values and key-value pairs, the system compares the values and key-value pairs to one or more knowledgebases that include: (1) regular expressions from existing parsers, (2) regular expressions for value types associated with normalized fields, and (3) a list of keys in key-value pairs associated with normalized fields. As the system learns new token-to-normalized fields relationships, the system adds the relationships to its knowledgebase.