Virtual Fields for Distributed Log Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current platforms for querying distributed log data lack support for complex operations and are resource-intensive, particularly as log messages become increasingly distributed and varied in format across multiple log endpoints.
Innovation Solution
The system enables users to define query strings that include virtual fields, which are user-defined data fields not present in log messages but can be referenced as functions or expressions of existing data fields, allowing for complex log analytics across multiple log endpoints regardless of message format, through a distributed processing layer that parses and aggregates data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If log messages are stored in various formats across multiple log endpoints, then the system can accommodate diverse data sources and formats, but querying becomes more complex and resource-intensive
Solution Approach 1:
The patent introduces a query translation layer that acts as an intermediary between the user's virtual field query and the actual log data storage. This translation layer converts virtual field references into actual field lookups, performing the function or expression of virtual fields on values returned from queries. This mediator handles format inconsistencies and distributed storage complexity, allowing users to query using abstract virtual fields without dealing with the underlying format diversity and distribution complexity.
2Adaptability or versatility
If virtual fields are implemented as user-defined functions or expressions, then complex log analytics can be performed, but processing resources increase
Solution Approach 1:
The system performs preliminary actions by pre-parsing log messages and pre-identifying data fields during the query preparation phase. The query translation layer analyzes the query string in advance, identifies virtual field definitions, and prepares the translation mapping before actual data retrieval. This preliminary processing organizes the work to be done, allowing the system to efficiently execute complex analytics functions without redundant processing during the actual query execution.
Solution Approach 2:
The query translation layer implements self-service by automatically analyzing query strings, identifying virtual field references, and translating them into executable queries without requiring manual intervention or complex configuration. The system autonomously handles the transformation of virtual field concepts into concrete data retrieval operations, reducing the processing burden on external systems and optimizing resource utilization during query execution.
3Quantity of substance
If log messages are distributed across multiple log endpoints, then data capacity and scalability increase, but query execution time increases
Solution Approach 1:
The patent applies segmentation by dividing the query execution process into distinct phases: query translation, data retrieval from distributed endpoints, and result aggregation. The query translation layer segments the virtual field query into component field references and translation rules, which are then applied during data retrieval. This segmentation allows parallel processing across multiple log endpoints while maintaining coordination, reducing overall query execution time despite the distributed nature of the data storage.
Data Source
AI summary
The present disclosure relates to querying log messages stored at log endpoints (e.g., log cores) using query strings that reference virtual fields, and generating log field views presenting aggregate information for groups of log messages that are grouped based on criteria defined in the query string.