Telemetry Table Merging by Epoch for Cluster Query Simplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cluster networks, the proliferation of separate tables for different metric streams related to the same resource complicates data extraction and querying, especially in streaming telemetry applications, due to the creation of multiple tables for replication pre-compression and replication network data.
Innovation Solution
A dynamic telemetry process that merges multiple streaming telemetry data streams for a specific resource into a single table using epochs and caching, allowing data to be aggregated around time boundaries and eliminating duplicates, thereby simplifying data storage and extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If separate tables are created for every metric data stream, then data storage is straightforward and simple, but the number of tables proliferates excessively making database querying very difficult
Solution Approach 1:
The patent merges multiple separate metric data streams into a single unified table structure. Instead of creating separate tables for each metric stream (e.g., replication precompression, replication network), the system consolidates them into one table with differentiated columns, thereby reducing table proliferation while maintaining organized storage of diverse telemetry data.
2Ease of manufacture
If multiple tables are created for different metric streams, then data organization is simple and direct, but data extraction becomes complicated across numerous tables
Solution Approach 1:
By consolidating multiple metric streams into a single table, the patent enables simplified data extraction operations. Users can query all relevant telemetry data for a resource in one table rather than joining multiple separate tables, significantly improving extraction efficiency while maintaining clear organization through structured columns and namespaces.
3Reliability
If every metric stream is stored as a separate table, then data integrity for each stream is maintained, but database querying becomes very difficult with excessive numbers of tables
Solution Approach 1:
The patent maintains data integrity by preserving distinct columns and data types for each metric stream within the unified table structure. Each metric stream retains its specific schema and validation rules, ensuring data integrity while the single-table approach eliminates the complexity of managing numerous separate tables for querying and management.
Data Source
AI summary
A telemetry processing system in a cluster network collecting streaming telemetry data from a plurality of telemetry producer pods. Processes improve extraction of telemetry data from a cluster network by collating streaming telemetry data received from each pod as a resource of the network, wherein a defined epoch delineates the streaming data into a plurality of metric datasets. Each dataset is stored in a database table created for the epoch in a respective cache created in a telemetry pipeline for each resource. The cached data is merged for a specific resource for the epoch in a merged database table that combines metric datasets per epoch to remove duplicative information created by the plurality of metric datasets and facilitate more efficient searching and extraction of the streaming telemetry data.