Statistical Time-Series Compression for Network Telemetry Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The exponential growth of data storage demands efficient and cost-effective solutions, as organizations face challenges with Petabytes of data and projected 175 Zettabytes by 2025, necessitating innovative compression methods to reduce storage needs.
Innovation Solution
Lossy statistical data compression encapsulates time-bounded data into mathematically modeled empirical functions, allowing for the deletion of redundant data and generation of representative functions, reducing storage requirements and costs, applicable in real-time or on-demand scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in its original form, then data integrity and completeness are maintained, but storage costs and storage space requirements increase exponentially
Solution Approach 1:
The patent extracts only the essential statistical characteristics and patterns from the original data, separating the core information needed for analysis from the redundant raw data. By extracting statistical features (mean, variance, distribution parameters) and storing only these compressed representations, the system achieves significant storage reduction while preserving the information necessary for most data analysis tasks.
Solution Approach 2:
The patent transforms raw data into different parameter representations through statistical analysis. Instead of storing original data points, the system stores statistical parameters (distribution parameters, trend coefficients, seasonal components) that characterize the data. This parameter transformation enables compact storage while maintaining the ability to reconstruct approximate data values and perform analytical queries.
2Productivity
If all empirical data is retained for analysis, then analytical accuracy is maximized, but processing time and computational resources increase
Solution Approach 1:
The patent segments the data processing task into two phases: an offline statistical analysis phase that computes and stores compressed statistical representations, and an online query phase that uses these pre-computed statistics for rapid analysis. This segmentation enables efficient real-time processing while maintaining analytical accuracy by performing heavy computations beforehand when computational resources are abundant.
Solution Approach 2:
The patent performs preliminary statistical analysis on the empirical data before actual querying occurs. By pre-computing statistical distributions, trends, and patterns and storing these results, the system eliminates the need to process raw data during query operations. This preliminary action significantly improves processing efficiency while preserving analytical accuracy through the pre-analyzed statistical representations.
Data Source
AI summary
A method performed in real-time includes receiving and storing time-based data over a specific time period and dividing the specific time period into a plurality of time windows. The method further includes determining that data associated with two or more proximate time windows are within a predetermined variance of one another and responsive to the determination: generating a mathematical function representative of the data associated with the two or more proximate time windows, deleting the data associated with the two or more proximate time windows, and generating a representation of the deleted data from the mathematical function. In certain embodiments, the data comprises empirical network telemetry data.