Geometric Aging Compression for ML Time-Series Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The accumulation of large volumes of time-series data from dense-sensor IoT applications poses challenges for storage and analysis, as existing compression algorithms often remove features essential for machine learning models, leading to suboptimal outcomes and increased storage costs.
Innovation Solution
The implementation of geometric aging data reduction (GAC) compresses time-series data based on age, preserving statistical features and patterns by averaging adjacent values, allowing for efficient storage and processing of large datasets without decompressing the data, thereby reducing storage overhead and processing times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression algorithms are applied to reduce storage overhead of time-series data, then storage costs are reduced, but statistical features and patterns are removed that are essential for machine learning models
Solution Approach 1:
The patent segments time-series data into fixed-size windows and applies different compression strategies to different segments. Recent data windows are stored at full resolution while older windows are compressed, allowing ML models to access high-quality features when needed while reducing overall storage overhead through selective compression of historical data.
Solution Approach 2:
The patent changes the resolution parameter of time-series data based on age. Recent data maintains high resolution with all original features, while older data is compressed to lower resolution. This parameter change allows the system to reduce storage overhead while preserving statistical features in the most recent data that are most critical for ML model training and inference.
2Measurement precision
If high sampling rates are used to transmit telemetry signals, then data accuracy is improved, but storage requirements and processing times increase significantly
Solution Approach 1:
The patent implements dynamic resolution adjustment where the sampling rate and data resolution are not fixed but adapt based on the age of the data. Recent data is stored at high sampling rates to capture transient events and maintain measurement precision, while older data is automatically downsampled to lower resolutions, significantly reducing storage requirements without losing critical information for current ML operations.
3Reliability
If all time-series data is stored at full resolution, then machine learning model accuracy is maintained, but storage costs and processing times become prohibitive
Solution Approach 1:
The patent segments the time-series database into multiple resolution levels organized in a pyramid structure. ML models can query data at the appropriate resolution level based on the specific task requirements, improving processing efficiency by avoiding unnecessary loading of high-resolution data when lower resolution suffices, while still maintaining access to full-resolution data when model accuracy requires it.
Solution Approach 2:
The patent applies partial compression by retaining full-resolution data only for the most recent time windows that are most relevant for current ML operations, while applying compression to older data. This partial action approach maintains model accuracy for recent patterns while improving processing efficiency by reducing the volume of historical data that needs to be processed and stored.
Data Source
AI summary
Techniques for geometric aging data reduction for machine learning applications are disclosed. In some embodiments, an artificial-intelligence powered system receives a first time-series dataset that tracks at least one metric value over time. The system then generates a second time-series dataset that includes a reduced version of a first portion of the time-series dataset and a non-reduced version of a second portion of the time-series dataset. The second portion of the time-series dataset may include metric values that are more recent than the first portion of the time-series dataset. The system further trains a machine learning model using the second time-series dataset that includes the reduced version of the first portion of the time-series dataset and the non-reduced version of the second portion of the time-series dataset. The trained model may be applied to reduced and/or non-reduced data to detect multivariate anomalies and/or provide other analytic insights.