Cluster Telemetry Delta Encoding for Lower Bandwidth Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cluster network systems experience significant network bandwidth consumption due to the repeated transmission of duplicative numeric and non-numeric telemetry data, which is not effectively optimized by existing solutions, particularly in time-series datasets.
Innovation Solution
Implementing a dynamic telemetry process that includes a subscription-based model with role-based access control, automatic security compliance checks, and a process for encoding duplicate data values to optimize network bandwidth, utilizing a new data instrument called 'change' to encode delta values for both numeric and non-numeric data sets during periodic epochs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If telemetry data is transmitted repeatedly from cluster components, then complete system monitoring is achieved, but network bandwidth consumption increases significantly
Solution Approach 1:
The patent extracts and transmits only the changed portions of telemetry data (delta values) rather than complete data sets. The encoding process identifies which metric values have changed between sampling intervals and transmits only those changes, eliminating redundant transmissions of unchanged data while maintaining complete system monitoring capability.
Solution Approach 2:
The system implements periodic sampling of telemetry metrics at defined intervals. By comparing current samples against previous samples and only transmitting when changes are detected, the system achieves continuous monitoring coverage through periodic actions rather than continuous transmission, reducing network bandwidth consumption while maintaining monitoring reliability.
2Loss of information
If all telemetry data is transmitted and stored, then complete data availability is achieved, but data storage and processing overhead increases
Solution Approach 1:
The encoding process extracts only the essential information - the delta values representing changes in metric states - and transmits this condensed data representation. This approach maintains complete data availability for monitoring purposes while significantly reducing the volume of data that needs to be stored and processed, as only changed values are retained rather than complete historical data sets.
3Productivity
If incremental backup methods are used for telemetry data, then transmission efficiency is improved, but applicability to time-series datasets is limited
Solution Approach 1:
The patent changes the fundamental parameter being transmitted from complete metric values to delta values (changes in values). This parameter transformation enables incremental transmission efficiency for time-series data while maintaining compatibility with open telemetry standards, as the encoding works specifically for time-series metric data where values change over time, making it adaptable to standard time-series processing pipelines.
Data Source
AI summary
A telemetry processing system in a cluster network collecting streaming telemetry data from a plurality of telemetry producer pods. Processes optimize network bandwidth by minimizing transmission of unchanged telemetry data within a defined epoch that delineates the streaming data into a plurality of metric datasets. New and previous time-series data sent by a pod are compared in a cache deployed in the pod. Data that is not changed raises a False Boolean value and is not stored by a telemetry pipeline. Data that is changed raises a True Boolean value and is stored in a datastore with the new data values inserted into a database stored in a datastore.