Telemetry Data Augmentation via Pre-computed Metrics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing data from computer networks for IT security, operations, and compliance is challenging due to its voluminous and rapid generation, and the data is often restricted, making it difficult and expensive to process effectively, especially when answering metrics-related questions at query time from log data that requires repeated parsing and transformation.
Innovation Solution
Transforming telemetry data into metrics time series and performing roll-ups to reduce the information processed, allowing for efficient querying and pre-computation, thereby optimizing query time and enabling efficient analysis and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If log data is stored and queried in its original format, then complete information is preserved, but query time and processing cost increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing aggregated metrics from log data before queries are executed. The system transforms raw log data into pre-aggregated metrics at data ingestion time, so that subsequent queries can retrieve results without performing repeated parsing and aggregation operations, thus reducing query time while preserving information through pre-computed aggregates
Solution Approach 2:
The patent creates a copy of the data in a transformed format - converting raw log data into metrics data with aggregated values. This copying approach allows the system to store both the original log data and the transformed metrics data, enabling fast queries on the metrics copy while the original logs remain available for complete information retrieval when needed
2Measurement precision
If repeated parsing and transformation of log data is performed at query time, then accurate metrics are obtained, but processing cost and time increase
Solution Approach 1:
The system performs the parsing and transformation operations in advance during data ingestion, converting raw logs into structured metrics data with pre-computed aggregates. This preliminary processing ensures measurement precision is maintained while eliminating the need for repeated transformations at query time, significantly improving processing efficiency
Solution Approach 2:
The patent changes the parameter representation of the data by transforming raw log entries into metrics with aggregated values and statistical parameters. This parameter transformation allows the system to retrieve pre-computed metrics directly without repeating parsing operations, maintaining accuracy while enhancing productivity
3Reliability
If all raw log data is stored for analysis, then complete data availability is maintained, but storage costs and data volume management become difficult
Solution Approach 1:
The patent extracts essential information from raw log data by computing aggregated metrics and statistical values. This extraction process creates a condensed representation of the data that maintains reliability for analysis purposes while significantly reducing the quantity of data that needs to be stored and managed
Solution Approach 2:
The system changes the data representation from detailed raw log entries to aggregated metrics with summary parameters. This parameter transformation reduces data volume by storing consolidated information (e.g., counts, averages, min/max values) instead of individual log events, while maintaining data availability for analytical purposes
Data Source
AI summary
Data enrichment and augmentation is disclosed. Machine data comprising at least one of a log message and a metrics data point is received. The received machine data comprises an identifier of an instance of a virtual machine. Based at least in part on the identifier of the instance of the virtual machine, a query for tags associated with the instance of the virtual machine is performed. At least one key-value pair is generated based at least in part on tags received in response to the query performed based at least in part on the identifier of the instance of the virtual machine. The received machine data is augmented with the at least one key-value pair generated based at least in part on the tags received in response to the query based at least in part on the identifier of the instance of the virtual machine.