Metrics Time Series Query Cardinality Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently handling the cardinality of ephemeral and high-volume data, particularly in monitoring and observability, as they struggle to optimize queries related to telemetry data delivered in log form, leading to costly and time-consuming parsing and transformation processes.
Innovation Solution
The transformation of telemetry data into metrics time series, combined with roll-ups, allows for efficient processing and querying by reducing the amount of data to be processed at query time, enabling faster response times and enabling pre-computation of structure-dependent operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If telemetry data is processed and queried in log form, then data completeness is maintained, but query time and processing cost increase significantly
Solution Approach 1:
The system performs preliminary transformation of telemetry data from log form to metrics time series form during data ingestion, before queries are executed. This pre-processing includes parsing logs, extracting metrics, and storing them in an optimized time series format with roll-ups, so that query operations can directly access pre-processed data without repeated parsing and transformation overhead
Solution Approach 2:
The system segments the data processing pipeline into distinct stages: log ingestion, metrics extraction, time series construction, and query execution. By maintaining both log data and transformed metrics data in separate but correlated structures, the system allows queries to operate on the optimized metrics representation while the original logs remain intact for completeness
2Productivity
If telemetry data is stored and queried with high cardinality, then data detail is preserved, but processing efficiency deteriorates
Solution Approach 1:
The system changes the parameter representation of telemetry data from high-cardinality log fields to normalized metrics with standardized time series parameters. By transforming data into a consistent metrics format with controlled cardinality through roll-up operations, the system improves processing efficiency while preserving the ability to retrieve detailed information when needed
3Ease of manufacture
If telemetry data undergoes parsing and transformation at query time, then query flexibility is maintained, but processing cost increases
Solution Approach 1:
The system performs parsing and transformation operations preliminarily during data ingestion rather than at query time. Metrics are extracted and structured into time series with roll-ups in advance, enabling queries to execute on pre-processed data. Query flexibility is maintained through the ability to specify different time ranges, aggregation levels, and metric types without triggering additional parsing overhead
Data Source
AI summary
Querying of time-aware metrics time series includes receiving a query, the query comprising a set of query metadata and a query time range. It further includes, based at least in part on the set of query metadata and the query time range, selecting a time series from a plurality of metrics time series. Each metrics time series in the plurality of metrics time series is associated with a set of metadata and an active interval of time. A set of metadata associated with the selected time series matches the set of query metadata, and an active interval of time associated with the selected metrics time series intersects with the query time range. The selected metrics time series is returned.