Time Series Data Reduction for Faster Outlier Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large sets of time series data in distributed computing systems are inefficiently stored and processed, leading to delayed detection of anomalies and increased storage costs, as existing management tools struggle with the computational bandwidth and data storage demands.
Innovation Solution
The method involves determining clusters and a normalcy domain in the time series data, quantizing data points within this domain, and compressing them while retaining outlier data points, which are then used to replace the original data, reducing storage needs and enabling faster analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large sets of time series data are stored and processed using existing management tools, then complete data retention is achieved, but storage costs increase and anomaly detection is delayed
Solution Approach 1:
The patent extracts and retains only outlier data points that deviate from the normalcy domain while compressing or discarding normal data points. This selective extraction maintains the critical information needed for anomaly detection while significantly reducing storage requirements and processing time, directly resolving the contradiction between complete data retention and timely anomaly detection.
Solution Approach 2:
The patent transforms the time series data by identifying and retaining only those data points that fall outside the normalcy domain (outliers), changing the parameter of data representation from storing all points to storing only anomalous points. This parameter change enables faster processing and reduced storage while maintaining anomaly detection capability.
2Loss of information
If large sets of time series data are stored in data-storage devices, then data completeness is maintained, but storage space is consumed and processing speed decreases
Solution Approach 1:
The patent extracts only the essential information (outlier data points) from the complete time series data while discarding redundant normal data points. This extraction maintains the critical information needed for anomaly detection while significantly reducing storage volume, resolving the contradiction between data completeness and storage efficiency.
Solution Approach 2:
The patent applies different treatment to different portions of the data: outlier data points are retained in full detail while normal data points are compressed or discarded. This local quality approach preserves critical anomalous information while reducing overall storage requirements.
3Reliability
If all time series data points are processed for anomaly detection, then detection accuracy is maximized, but computational resources are overwhelmed and processing is delayed
Solution Approach 1:
The patent extracts only the relevant data points (outliers) that are likely to contain anomalies, eliminating the need to process all data points. This extraction maintains detection accuracy by focusing on critical data while dramatically improving processing speed by reducing the data volume that must be analyzed.
Solution Approach 2:
Instead of processing all data points equally, the patent applies partial action by selectively processing only outlier data points that fall outside the normalcy domain. This partial processing approach maintains detection accuracy for anomalies while improving overall processing productivity.
Data Source
AI summary
Automated methods and systems to reduce the size of time series data while maintaining outlier data points are described. The time series data may be read from a data-storage device of a physical data center. Clusters of data points of the time series data are determined. A normalcy domain of the time series data and outlier data points of the time series data is determined. The normalcy domain of the time series data comprises ranges of values associated with each clusters of data points. The outlier data points are located outside the ranges. Quantized time series data are computed from the normalcy domain. When the loss of information due to quantization is less than a limit, the quantized time series data is compressed. The time series data in the data-storage device is replaced with the compressed time series data and outlier data points.