Dynamic Parsing Rules for Log Data Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management tools face challenges in normalizing unstructured log data from diverse sources, requiring different normalization rules for each device, operating system, and application, which complicates data analysis and retrieval.
Innovation Solution
The implementation of dynamic parsing rules, including system and custom parsing rules, utilizes a state machine to detect conflicts and distribute rules across a hierarchical parsing system, enabling flexible and scalable data normalization and storage in a centralized data store.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If different normalization rules are created for each device, operating system, and application, then data from diverse sources can be normalized, but the system complexity increases significantly
Solution Approach 1:
The patent segments the normalization process into multiple tiers: a first-tier parser identifies log source types, and second-tier parsers handle specific normalization rules for each type. This segmentation allows the system to manage diverse data sources without requiring a single complex rule set, thereby reducing overall system complexity while maintaining adaptability.
Solution Approach 2:
The patent creates a universal first-tier parser that can identify various log source types through common patterns. This universal component handles the initial classification for all diverse sources, eliminating the need for separate identification logic for each device, operating system, or application, thus reducing complexity while preserving normalization capability.
2Reliability
If a centralized system processes all log data, then data consistency is maintained, but processing performance and scalability are limited
Solution Approach 1:
The patent divides the processing system into centralized rule management and distributed parsing execution. The first-tier parser centrally identifies log types and routes data appropriately, while second-tier parsers are distributed across different processing nodes. This segmentation enables both data consistency through centralized routing and high throughput through distributed parallel processing.
Solution Approach 2:
The patent adds a hierarchical dimension to the processing architecture, with first-tier parsing operating at one level (log source identification) and second-tier parsing operating at another level (specific data normalization). This multi-dimensional approach allows simultaneous centralized control for consistency and distributed execution for scalability.
3Speed
If parsing rules are rigidly defined, then processing speed is maintained, but flexibility to adapt to new log formats is reduced
Solution Approach 1:
The patent implements dynamic parsing rules in the second tier that can be adjusted based on identified log source types. The system maintains fast processing through pre-compiled first-tier parsing rules for log type identification, while allowing flexible, dynamically configurable second-tier rules to adapt to specific formats and new log types without sacrificing overall processing speed.
Data Source
AI summary
Methods, program products, and systems implementing dynamic parsing rules are disclosed. Log data from a variety of log producers can be parsed using parsing rules to generate information about an information system. The parsing rules can include system parsing rules and custom parsing rules. A state machine can be used to detect conflicts between various parsing rules. A central server can distribute the system parsing rules and custom parsing rules to one or more remote servers for distributed processing. In a hierarchical parsing system, a first tier parser can be used to identify types of sources generating the log data. Log data from each type of log source can be sent to a second tier parser that corresponds to the type of log source.