Iterative Outlier Detection via Mean Deviation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional K-Nearest Neighbor (KNN) algorithm requires user input for the number of outliers to be removed, which may not accurately detect the actual number of outliers in a data set, especially when the user is unaware of this information, leading to incorrect outlier classification.
Innovation Solution
An iterative method that calculates Euclidean distances, average distances, and deviations to automatically detect outliers by sorting and iteratively refining the list of values based on mean and average deviation, eliminating the need for user-defined parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional KNN algorithm is used with user-defined parameters, then the algorithm can be executed, but the accuracy of outlier detection deteriorates when user input is incorrect or unavailable
Solution Approach 1:
The system performs self-service by automatically determining the number of outliers to detect through iterative calculation of mean and average deviation, eliminating the need for user input about the number of outliers while maintaining detection accuracy
Solution Approach 2:
The algorithm dynamically adjusts parameters (mean and average deviation) through iterative refinement, changing the threshold criteria for outlier classification based on calculated statistics rather than fixed user-defined values
2Measurement precision
If iterative refinement with mean and average deviation is implemented, then outlier detection accuracy is improved, but computational complexity increases
Solution Approach 1:
The algorithm employs dynamic iterative refinement where the list of outliers is repeatedly updated based on recalculated mean and average deviation, allowing the system to adapt and improve accuracy through multiple passes rather than using a static single-pass approach
Data Source
AI summary
An iterative approach to detect outliers may automatically detect outliers in a data set. The iterative approach may include calculating a mean and an average deviation for a list. The iterative approach may further include calculating the deviations of values of a list if the average deviation is not zero. Any values where the calculated deviation of the point or value is greater than the mean may be selected. The selected points may be set as a second list. The second list may be used by the iterative approach until no points are selected for the second list. The previous list is the set of outliers from the data set.