Metric Data Compression Using Dynamic Dictionaries and Time Deltas
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional metric compression systems are inflexible and inefficient, often requiring large memory and processing resources due to static compression dictionaries and rigid compression methods, leading to significant bandwidth consumption for updates and limited temporal compression capabilities.
Innovation Solution
A graph-based compression dictionary and time slice delta compression are employed to dynamically update nodes and edges, generating flexible and efficient compressed metric data by using a finite state transducer to traverse nodes and edges, and applying delta compression across time slices and windows.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional static compression dictionaries are used, then system simplicity is maintained, but processing power and memory resources are excessively consumed
Solution Approach 1:
The patent transforms static compression dictionaries into dynamic structures that adapt to changing data patterns. The system automatically updates dictionary entries based on observed metric relationships, allowing the compression model to evolve and improve efficiency without requiring excessive memory resources. This dynamic adaptation resolves the contradiction by making the system responsive to actual data characteristics rather than relying on fixed, resource-intensive pre-computed dictionaries.
Solution Approach 2:
The system changes the parameters of the compression dictionary over time by introducing temporal dimensions and delta compression techniques. Instead of using fixed compression ratios and dictionary sizes, the system adjusts these parameters dynamically based on the temporal patterns observed in metric data, thereby improving compression efficiency while maintaining manageable memory and processing requirements.
2Adaptability or versatility
If conventional rigid compression methods are used, then implementation simplicity is maintained, but adaptability to different metric patterns is limited
Solution Approach 1:
The patent implements dynamic compression methods that automatically adapt to different metric patterns through continuous learning and adjustment. The system observes temporal relationships in metric data and dynamically modifies compression strategies, enabling high adaptability to varying metric patterns without requiring complex manual configuration or multiple specialized compression algorithms.
Solution Approach 2:
The system creates a universal compression framework that handles multiple types of metric patterns through a single adaptive mechanism. By using delta compression and temporal pattern recognition, the system can effectively compress diverse metric types (counters, gauges, histograms) without requiring separate specialized methods for each type, thereby achieving versatility without proportionally increasing system complexity.
3Productivity
If large compression dictionaries are used, then compression ratio is improved, but bandwidth consumption for updates increases significantly
Solution Approach 1:
The patent employs delta compression techniques that change the representation parameters from absolute values to relative differences. By compressing the difference between consecutive metric values rather than the values themselves, the system achieves high compression ratios while transmitting significantly less data, thereby reducing bandwidth consumption for dictionary updates and metric transmissions.
Solution Approach 2:
The system extracts only the essential changing information from metric data by using delta compression. Instead of transmitting or storing complete metric values and full dictionary updates, the system extracts and transmits only the differences or changes, thereby achieving effective compression with minimal bandwidth consumption while maintaining the ability to reconstruct complete information at the receiving end.
4Loss of time
If conventional compression systems are used, then temporal compression is limited, but system complexity is reduced
Solution Approach 1:
The patent implements periodic action by organizing compression around time slices and temporal patterns. The system processes metric data in periodic time intervals, applying delta compression that exploits the temporal regularity of metric updates. This periodic approach enables effective temporal compression by recognizing and compressing repeating patterns across time periods while maintaining a relatively simple and systematic compression methodology.
Solution Approach 2:
The system performs preliminary action by pre-computing and storing delta relationships and temporal patterns before actual compression is needed. By establishing the temporal compression framework in advance and pre-processing metric relationships, the system enables efficient real-time compression without requiring complex runtime computations, thereby achieving superior temporal compression with manageable system complexity.
Data Source
AI summary
A computing system configured to (i) obtain a set of key-value pairs, wherein each key- value pair corresponds to a respective timestamp in a period of time, (ii) for at least one timestamp in the given period of time: (a) identify a first subset of the key-value pairs corresponding to the timestamp, (b) sort the first subset, and (c) generate a subset of compression values for the sorted first subset, (iii) for at least one key: (a) identify a second subset of the key-value pairs corresponding to the key, (b) sort the second subset, and (c) generate a subset of compression values for the sorted second subset, and (iv) store a set of compression values comprising (a) the subset of compression values that is generated for each of the at least one timestamp and (b) the subset of compression values that is generated for each of the at least one key.