Hybrid Data Sequence Compression for Precision Metric Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing APM systems inefficiently handle large quantities of data by continuously sending metrics that remain unchanged or follow a pattern, leading to wasteful use of network resources and potential loss of data precision.
Innovation Solution
A method for transforming datapoints in a first data sequence to a second data sequence, where datapoints fitting a pattern are expressed as an expression in a hybrid bounded and unbounded data sequence, reducing storage and network usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If metrics are continuously sent to APM backend at regular intervals, then data precision is maintained, but network resources are wasted and storage requirements increase
Solution Approach 1:
The patent extracts only the necessary data points from the continuous metrics stream - specifically sending data only when values change or when bounded sequences are broken. This removes redundant data transmission while preserving essential information for monitoring and analysis.
Solution Approach 2:
The system dynamically adjusts the data transmission strategy based on the detected patterns. Bounded sequences trigger expression-based compression, while unbounded sequences trigger immediate transmission. This dynamic adaptation optimizes network usage while maintaining data precision where needed.
2Reliability
If all datapoints are transmitted and stored, then complete data availability is ensured, but storage space consumption increases
Solution Approach 1:
The patent changes the representation parameters of the data by transforming sequences of datapoints into mathematical expressions. This parameter transformation compresses the data representation from individual numeric values to compact algebraic formulas, significantly reducing storage requirements while maintaining data availability for reconstruction.
Solution Approach 2:
Instead of storing all original datapoints, the system stores compact expressions that can be evaluated to reproduce the data values. This copying approach using mathematical representations reduces storage space while ensuring data availability when needed.
3Loss of energy
If data compression is applied to reduce storage and network usage, then resource efficiency improves, but data precision may be lost
Solution Approach 1:
The system uses feedback mechanisms to verify that compressed expressions accurately represent the original data sequences. By monitoring whether data points fit bounded patterns and can be precisely expressed, the system ensures that compression does not compromise data precision before applying the compression.
Solution Approach 2:
The patent segments the data stream into bounded and unbounded sequences, applying different processing strategies to each segment. This segmentation allows precise compression for predictable patterns while maintaining full precision for unpredictable data, thereby preserving overall data accuracy while improving efficiency.
4Loss of energy
If manual configuration of metric collection intervals is used, then data transmission can be reduced, but data precision is lost due to larger intervals
Solution Approach 1:
The system performs self-service by automatically detecting patterns in the data stream and dynamically adjusting its transmission strategy without manual configuration. It autonomously identifies bounded sequences for compression and unbounded sequences for immediate transmission, eliminating the need for manual interval setting while maintaining both efficiency and precision.
Data Source
AI summary
An embodiment includes detecting a first data sequence by a system. The embodiment includes responsive to the detecting, determining by the system in real time whether a number of datapoints in the first data sequence is expressible as an expression. The embodiment also includes responsive to the determining, deciding to transform a datapoint in the first data sequence to the expression in a second data sequence wherein the first data sequence is transformed to the second data sequence.