Anomaly Detection in Time Series Data via Sliding Window Statistics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods are inefficient in quickly recognizing and characterizing anomalous elements, trends, and patterns within large volumes of one-dimensional data, such as time series and spectral data, which is particularly challenging in large-scale data sets like those generated by DOCSIS 3.1 cable modems.
Innovation Solution
The method involves smoothing and detrending the data series, then using a sliding window to calculate the median and standard deviation. Anomalous elements are identified when the standard deviation exceeds a threshold, and the data series is scored based on the quantity and magnitude of these anomalies. This process can be repeated with modified data series to further refine anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional data analysis methods are used on large data sets, then comprehensive analysis can be performed, but the time and resources required become cost-prohibitive
Solution Approach 1:
The patent segments the large data set into smaller manageable chunks using a sliding window approach. Each window processes a subset of data points independently, calculating local statistics (mean, standard deviation) to identify anomalies. This segmentation enables parallel processing and reduces the computational burden on any single processing unit, thereby decreasing overall analysis time while maintaining detection accuracy through systematic coverage of the entire data set.
2Measurement precision
If traditional data analysis methods are used on large data sets, then comprehensive analysis can be performed, but the resources required become cost-prohibitive
Solution Approach 1:
The patent extracts only the essential features needed for anomaly detection from the raw data. By calculating simple statistical measures (mean, standard deviation) within sliding windows and comparing these to threshold values, the method extracts anomaly information without performing computationally intensive operations. This extraction approach maintains detection accuracy while significantly reducing computational resource requirements compared to comprehensive analysis methods.
3Reliability
If all data samples are analyzed, then no anomalies are missed, but the majority of good data samples increase processing time unnecessarily
Solution Approach 1:
The patent applies local quality analysis by examining data characteristics within localized sliding windows rather than analyzing the entire data set uniformly. Each window is evaluated based on its own local statistics (mean, standard deviation), allowing the method to adapt to local variations in the data. This approach ensures that anomalies are detected with high reliability in each local region while avoiding the inefficiency of processing all data points with the same computational intensity, thereby improving overall processing efficiency.
Data Source
AI summary
The present disclosure generally relates to apparatus, software and methods for detecting anomalous elements in data. For example, the data can be any time series, such as but not limited to radio frequency data, temperature data, stock data, or production data. Each type of data may be susceptible to repeating phenomena that produce recognizable features of anomalous elements. In some embodiments, the features can be characterized as known patterns and used to train a machine learning model via supervised learning to recognize those features in a new data series.