Recursive Sub Parsing for Heterogeneous Log Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional log analytics tools face challenges in efficiently collecting, analyzing, and parsing large volumes of log records from numerous computing systems and applications, particularly due to the complexity of wrapped log records in various formats, leading to redundant processing and resource inefficiencies.
Innovation Solution
A system and method that employs multiple parsers to configure, collect, and analyze log records by identifying base parsers based on log record types, using field mappings to associate base field values with sub-parsers, and merging sub-fields with base fields to generate interpretable output, enabling efficient parsing and analysis of wrapped log records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple parsers are configured to handle wrapped log records in various formats, then the system can accurately parse and extract meaningful data from heterogeneous log sources, but the device complexity and configuration overhead increase significantly
Solution Approach 1:
The log parsing system is divided into multiple independent parsers, each specialized for a specific log format (JSON, XML, plain text, etc.). Each parser is configured separately with its own field mappings and extraction rules, allowing the system to handle diverse log formats without requiring a single complex monolithic parser. This segmentation enables precise parsing for each format while maintaining manageable configuration complexity through modular design.
2Ease of operation
If conventional per-host log collection approach is used, then each host can be configured and analyzed individually, but the system cannot efficiently scale to large numbers of hosts and applications
Solution Approach 1:
The log analytics system is designed with universal parsers and field mappings that can process log records from multiple hosts and applications simultaneously. The system extracts common fields (timestamp, hostname, application, log level) and format-specific fields using the same parsing infrastructure, enabling it to scale efficiently across large numbers of hosts without requiring separate configuration for each individual host, while still maintaining the ability to handle individual host-specific log formats.
3Ease of manufacture
If JSON-based parsing mechanism is used to unwrap wrapped logs, then the outer envelope can be parsed, but the inner payload remains JSON-escaped and cannot be parsed further
Solution Approach 1:
The system performs preliminary detection of the inner payload format within the JSON-escaped string during the parsing configuration phase. Field mappings are pre-configured to identify format indicators (JSON brackets, XML tags, plain text patterns) within the escaped payload. Based on this preliminary detection, the system automatically selects and applies the appropriate inner parser (JSON, XML, or plain text parser) to further parse the payload, preventing loss of parseability information.
4Loss of information
If comprehensive parsing of all log fields is performed, then complete log data can be extracted and analyzed, but computational resources and processing time increase
Solution Approach 1:
The parsing system applies local quality by configuring different levels of parsing depth and field extraction based on specific log sources and analysis requirements. Field mappings are selectively defined to extract only the most relevant fields for each log type and analysis scenario. The system can configure parsers to extract only essential fields (timestamp, hostname, message) for high-volume logs, while extracting comprehensive fields for critical application logs, thereby optimizing computational resource usage while maintaining data completeness where needed.
Data Source
AI summary
A system and computer-implemented method for a log analytics system that can configure, collect, parse, and analyze log records in an efficient manner. Log records are accessed, each of the log records is associated with a log source. A base parser is identified for parsing a log record based on a type of the log record indicated in the log source. The log record is parsed using the base parser to extract base field values corresponding to base fields. A base-parsed log record is generated on parsing. Sub-parsers are identified using field mappings. The field mappings include base field values mapped to corresponding sub-parsers. The base-parsed log record is parsed using the sub-parsers to extract sub-fields. The sub-fields are merged to the base fields to generate and present an output that includes the parsed log record, the base fields, base field values, the sub-fields and the sub-field values.