Date Outlier Detection Using Median Absolute Deviation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analytics systems face challenges in accurately identifying and updating date values in datasets to support effective monitoring and reporting, particularly in identifying outliers and maintaining consistent date formats across different data sets.
Innovation Solution
A method is implemented where date values in a dataset are identified as outliers or non-outliers by computing median and median absolute deviation values of neighboring date values, comparing them to a threshold, and updating non-outlier date values using a determined date offset, ensuring consistent date formatting and alignment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional outlier detection methods are used to identify date values, then the process is simpler, but the accuracy of outlier detection deteriorates
Solution Approach 1:
The patent introduces an intermediary statistical framework using median and median absolute deviation (MAD) calculations as mediators between raw date values and outlier identification. This intermediary process transforms direct comparison into a multi-step statistical evaluation, improving detection accuracy by considering the distribution characteristics of neighboring date values rather than relying on simple threshold comparisons
Solution Approach 2:
The patent applies preliminary action by calculating the median and MAD of neighboring date values before performing outlier detection. This preparatory statistical computation establishes a dynamic baseline for each date value, enabling more accurate outlier identification by accounting for local data patterns rather than using fixed global thresholds
2Stability of the object's composition
If date values are updated without outlier detection, then the process is faster, but the consistency of date formats across datasets deteriorates
Solution Approach 1:
The patent applies local quality by treating each date value individually with its own outlier assessment based on neighboring values. Instead of applying a uniform update rule to all dates, the system evaluates each date's local context through median and MAD calculations, determining whether it should be updated based on its specific position within the date distribution. This localized approach maintains format consistency while preserving legitimate date variations
Solution Approach 2:
The patent extracts outlier date values from the dataset before applying bulk date updates. By identifying and separating outliers through statistical analysis, the system applies updates only to non-outlier values, preventing the propagation of inconsistent date formats while maintaining the integrity of legitimate date variations in the extracted outlier group
3Reliability
If all date values are updated uniformly, then the reporting consistency is improved, but the accuracy of temporal analysis deteriorates due to outlier distortion
Solution Approach 1:
The patent converts the potential harm of outlier date values into a benefit by using them as reference points for statistical analysis. Rather than simply removing or correcting outliers, the system leverages their presence to establish meaningful baselines through median and MAD calculations. This approach transforms outlier distortion into an opportunity for more robust temporal analysis that accounts for actual data variability rather than assuming uniform patterns
Data Source
AI summary
A computing device updates date values in a read dataset to support data analytics. Outlier and non-outlier date values are identified by, for each date value as a respective date value, reading a predefined number of neighboring date values relative to the respective date value; computing a median value and a median absolute deviation value of the predefined number of neighboring date values; computing a difference between the respective date value and the median value; dividing an absolute value of the difference by the median absolute deviation value to define a deviation value; comparing the deviation value to a threshold deviation value; and, based on the comparison, identifying the respective date value as an outlier or a non-outlier date value. Each identified non-outlier date value is updated with a new date computed using a date offset value. Each updated, identified non-outlier date value is replaced in a date updated dataset.