Telemetry Data Compression Using Polynomial Slice Fitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transmission of large volumes of telemetry data from computing environments with many sensors can overwhelm networks and processing systems, leading to inaccurate analysis due to down sampling and aggregation, which fails to capture transient or subtle fluctuations in performance.
Innovation Solution
Applying polynomial fitting, such as Chebyshev polynomials, to divide telemetry data into slices and use optimized slice sizes and fitting orders to compress data, transmitting only polynomial coefficients for reconstruction at the processing system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If telemetry data is transmitted at full resolution and frequency, then data integrity and accuracy are maintained, but network bandwidth and processing system resources are overwhelmed
Solution Approach 1:
The telemetry data is divided into multiple segments or chunks, with critical high-resolution data separated from less critical data. This allows selective transmission of essential information at full resolution while aggregating or reducing less critical data, balancing data integrity with network throughput
Solution Approach 2:
The patent dynamically adjusts data transmission parameters such as resolution, sampling frequency, and aggregation level based on network conditions, device priorities, and anomaly detection. Critical devices maintain high-resolution transmission while non-critical devices use aggregated data, optimizing both data integrity and network efficiency
2Productivity
If down sampling and aggregation are applied to reduce data volume, then network and processing burdens are reduced, but transient and subtle performance fluctuations are lost
Solution Approach 1:
Different levels of data processing and aggregation are applied to different devices or data streams based on their criticality. Devices identified as critical maintain high-resolution data transmission, while non-critical devices use aggregated data, ensuring detection accuracy for important metrics while improving processing efficiency overall
Solution Approach 2:
The system applies aggressive aggregation to non-critical data streams where some loss of detail is acceptable, while maintaining partial high-resolution data for critical parameters. This partial application of aggregation preserves essential transient fluctuations while reducing overall data volume for efficient processing
3Productivity
If polynomial fitting compression is applied to reduce data transmission, then network bandwidth is conserved, but computational complexity increases at the processing system
Solution Approach 1:
Polynomial fitting compression is applied at the data collection edge before transmission, performing the computationally intensive compression operation in advance. This shifts the computational burden from the central processing system to distributed edge devices, improving network efficiency while managing processing complexity through distributed computation
Data Source
AI summary
In some examples, a system selects a slice size and a polynomial fitting order from a plurality of candidate slice sizes and a plurality of candidate polynomial fitting orders for representing a series of telemetry data obtained by sensors in a computing environment, where the selecting is based on solving an optimization problem comprising variables representing the slice size, the polynomial fitting order, and a measure of fit. The system divides the series of telemetry data into a plurality of slices having the selected slice size, and applies compression of telemetry data in a slice of the plurality of slices using polynomial fitting according to the selected polynomial fitting order, the applied compression producing compressed telemetry data. The system transmits, over a network, the compressed telemetry data to a processing system for processing of the compressed telemetry data.