Outlier Removal Using Covariance and Density Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Big data processing systems face challenges in estimating effective processing time due to large variations in execution times, primarily because existing outlier detection techniques require known statistical data distributions, making it difficult to identify and remove outliers and determine meaningful statistics, especially when the data distribution is unknown.
Innovation Solution
A system that uses a combination of covariance estimation and density-based clustering algorithms to iteratively remove outliers from data sets, determining the type of distribution and applying appropriate methods to achieve a cleaned data set with a coefficient of variation within a specified threshold, effectively handling both normal and non-normal distributions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing outlier detection techniques are used, then outliers can be identified and removed, but the technique requires known statistical data distribution which is not available in many big data processing systems
Solution Approach 1:
The system changes the approach from assuming a specific data distribution to using non-parametric methods that adapt to the actual data distribution. By using techniques like modified z-score and interquartile range that do not require assuming normal distribution, the system can identify outliers regardless of the underlying data distribution characteristics.
Solution Approach 2:
The system performs self-adaptation by automatically detecting the data distribution characteristics and selecting appropriate outlier detection methods without external intervention. The algorithm dynamically adjusts its behavior based on the data it receives, making it self-sufficient for handling various distribution types.
2Measurement precision
If a large number of executions are collected to get meaningful statistics, then accurate statistics can be obtained, but the processing time varies significantly due to outlier data
Solution Approach 1:
The system performs preliminary outlier detection and removal before computing statistics. By identifying and excluding outliers in advance using distribution-free methods, the system ensures that subsequent statistical computations are based on clean data, improving accuracy without requiring excessive numbers of executions.
Solution Approach 2:
The system extracts and removes outlier data points from the dataset before performing statistical analysis. This extraction process uses robust statistical methods that do not require knowing the underlying distribution, thereby cleaning the data and enabling more accurate and efficient statistics computation.
3Quantity of substance
If outlier data is included in computations, then all data is utilized, but the statistics become skewed and meaningless
Solution Approach 1:
The system applies different handling to different data points based on their quality. By locally identifying and treating outliers differently from normal data points, the system maintains the usefulness of legitimate data while excluding skewed values that would compromise statistical validity.
Data Source
AI summary
A system for outlier detection and removal comprises an interface and a processor. The interface is configured to receive a data set. The processor is configured to determine a cleaned data set by removing outliers, wherein determining the cleaned data set comprises determining a type of distribution, in response to the type of distribution being normal, determining the outliers using covariance estimation, in response to the type of distribution not being normal, determining the outliers using density based clustering, and determining the cleaned data set by removing the outliers from the data set, determine a coefficient of variation of the cleaned data set, determine whether the coefficient of variation is greater than a threshold coefficient of variation, and in response to the coefficient of variation being greater than the threshold coefficient of variation, determine a new cleaned data set by removing a new set of outliers from the cleaned data set.