Structured Log Record Enrichment Through Header-Detail Parsing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional log analytics tools struggle to efficiently scale and analyze large volumes of log records across numerous computing systems and applications due to their per-host configuration requirements, inefficient resource usage, and inability to associate structured and unstructured log messages effectively, leading to redundant processing and lack of contextual information.
Innovation Solution
A method is introduced to create a header-details relationship among structured log records using two parsers, one for header and one for detail information, allowing enrichment of detail records with header fields, thereby associating and enriching log entries with contextual information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional log analytics tools are used to collect and analyze log records, then basic log collection and analysis capabilities are provided, but the system cannot efficiently scale when faced with massive systems involving large numbers of computing systems and applications
Solution Approach 1:
The patent segments log records into header records and detail records, allowing the system to process common information once and reuse it across multiple detail records. This segmentation enables efficient scaling by avoiding redundant processing of common fields across numerous computing systems and applications.
Solution Approach 2:
The patent creates a universal log analysis system that can handle multiple types of log records from different computing systems and applications through a unified header-details relationship model. This multi-functionality allows a single system to scale across diverse environments without requiring separate processing logic for each system type.
2Ease of operation
If per-host configuration approach is used in conventional systems, then log collection can be configured for each host, but extensive redundant processing and resource usage occur given the large number of hosts
Solution Approach 1:
The patent merges common log fields into a single header record that can be shared across multiple detail records from different hosts. Instead of configuring and processing each host separately, the system combines common information once at the header level and references it across all detail records, eliminating redundant processing while maintaining configuration flexibility.
Solution Approach 2:
The patent extracts and processes common log fields in advance during header record processing, before detail records are handled. This preliminary action of extracting common information once and caching it allows subsequent detail records to be processed more efficiently without repeating the same extraction operations for each host.
3Measurement precision
If structured log format is used, then consistent format and keys enable reliable extraction, but unstructured log messages lack this consistency making value extraction difficult
Solution Approach 1:
The patent segments log processing into header and detail components, where the header portion maintains structured format with consistent keys for reliable extraction, while the detail portion can accommodate varied log formats from different sources. This segmentation allows the system to enforce structure where needed while maintaining flexibility where log formats differ.
Solution Approach 2:
The patent applies local quality by requiring structured format with consistent keys specifically for header fields that need reliable extraction, while allowing detail fields to have varied formats appropriate to their specific log sources. This localized structuring maintains extraction reliability for common fields while preserving adaptability for diverse detail formats.
4Loss of information
If header-details relationship is created among structured log records, then contextual information is enriched to detail records, but system complexity increases due to two parsers and relationship management
Solution Approach 1:
The patent introduces an intermediary mechanism that manages the relationship between header and detail records through a standardized interface. This intermediary layer handles the complexity of matching and enriching records while keeping the overall system architecture manageable, allowing contextual information to be added without proportionally increasing system complexity.
Data Source
AI summary
Embodiments relate to extracting various portions of log messages. Syntax, order and/or level information that differentiates header and detail message is identified. This syntax, order and/or level information is used to automatically associate enhance each of one or more detail log entries with information that provides context. This approach can facilitate efficient transmission of detail-information log entries that reduces redundant information, while still supporting flexible approaches for providing information that can enhance detail-information log entries.