Hampel Filtering Outlier Detection Time Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Hampel filtering processes for outlier detection in time series data are computationally expensive due to sorting operations, and they struggle with handling missing data and non-Gaussian distributions.
Innovation Solution
An improved Hampel filtering process that optimizes the collaboration between computational steps to reduce time complexity from O(n(m log(m))) to O(nm), allowing for efficient handling of missing data and supporting multiple distribution types, including non-Gaussian distributions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional Hampel filtering processes are used for outlier detection, then the method can detect outliers in time series data, but the computational time complexity is high (O(n(m log(m))))
Solution Approach 1:
The patent segments the computational process into distinct phases: (1) computing the sliding window median, (2) computing absolute deviations from the median, (3) sorting only the absolute deviations, and (4) computing the median of the sorted absolute deviations. This segmentation allows each phase to be optimized independently, reducing the overall complexity from O(n(m log(m))) to O(nm).
Solution Approach 2:
The patent performs preliminary computation of the sliding window median and absolute deviations before sorting. By pre-computing these values, the algorithm avoids repeated sorting operations and reduces the sorting step to only the necessary sorted absolute deviations, achieving O(nm) complexity instead of O(n(m log(m))).
2Adaptability or versatility
If conventional Hampel filtering is applied, then outlier detection can be performed, but the method struggles with handling missing data and non-Gaussian distributions
Solution Approach 1:
The patent changes the parameter used for outlier detection from assuming Gaussian distribution to using the Median Absolute Deviation (MAD), which is robust to non-Gaussian distributions and missing data. This parameter change allows the algorithm to adapt to various data types and distributions without requiring assumptions about the underlying data generation process.
Data Source
AI summary
A new value is written from a dataset to a data structure comprising a set of sorted values. The new value replaces an oldest value and is inserted in a sorted position. The data structure is modified by subtracting a median value from each value of the set of sorted values to obtain sorted signed deviation values. The sorted signed deviation values are segmented to obtain data substructures comprising subsets of sorted absolute deviation values. A binary search is performed on the data substructures to identify a median absolute deviation value. A difference is computed between a particular value and the median value, and based on whether the difference is less than a threshold value computed from the median absolute deviation value, an outlier decision output is generated indicative of whether the particular value comprises an outlier value.