Structured Log Clustering by Key-Value Schema
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log analysis systems, designed for unstructured data, face challenges in efficiently analyzing and querying structured log data due to performance issues and incorrect structure inference, particularly with JSON logs, leading to incorrect results and scalability problems.
Innovation Solution
The development of structured log analysis techniques that cluster data by key schema and values using operators like LogReduce Keys and LogReduce Values, which are aware of the key-value structure, enabling correct matching and alignment of log data regardless of position, and facilitate improved analysis and querying of structured logs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing string-based log analysis systems are used to analyze structured log data, then the systems can process unstructured data with which they are familiar, but they produce incorrect results and suffer from performance degradation due to incorrect structure inference
Solution Approach 1:
The patent segments structured log data into distinct key-value pairs, separating the structure (keys) from the content (values). This segmentation allows the system to correctly identify and process the hierarchical structure of JSON logs, preventing the performance degradation and incorrect results that occur when treating structured data as unstructured strings.
Solution Approach 2:
The patent changes the fundamental parameter of how log data is represented - from treating logs as unstructured strings to representing them as structured key-value maps. This parameter change enables the system to properly infer structure, improving both the accuracy of analysis results and the performance of the system when processing structured log data.
2Adaptability or versatility
If existing log analysis systems attempt to infer structure from structured log data, then they can work with the data in a unified manner, but they produce incorrect structure inference leading to incorrect matching and alignment
Solution Approach 1:
The patent applies preliminary action by pre-processing log data to extract and preserve its key-value structure before analysis. Instead of attempting to infer structure during analysis, the system prepares the data by identifying keys and values upfront, ensuring precise structure recognition and correct matching/alignment throughout the analysis process.
Solution Approach 2:
The patent introduces an intermediary representation - a key-value map structure - that sits between the raw structured log data and the analysis processes. This intermediary preserves the hierarchical structure of JSON logs while providing a unified format for analysis, enabling both adaptability to structured data and precision in structure inference.
3Device complexity
If string-based systems process structured log data, then they can maintain simplicity in implementation, but they suffer from scalability problems and performance issues with large volumes of structured data
Solution Approach 1:
The patent substitutes the mechanical string-based processing system with a structured data processing approach. By replacing string manipulation with key-value map operations, the system maintains implementation simplicity while achieving scalability, as the structured approach enables efficient indexing, filtering, and aggregation operations on large volumes of log data.
Data Source
AI summary
Clustering structured log data by key-values includes receiving, via a user interface, a request to apply an operator to cluster log messages according to values for keys associated with the request. At least a portion of each log message comprises structured machine data including a set of key-value pairs. The method further includes receiving a log message and determining whether to include the log message in a cluster based at least in part on an evaluation of values in the structured machine data of the log message for the keys associated with the request. The cluster is included in a set of clusters. Each cluster in the set is associated with a different combination of values for the keys associated with the request. The method further includes providing, via the user interface, information associated with the cluster.