Outlier Filtration via PCA and Quantile Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning techniques for outlier detection in large datasets struggle with complex or high-dimensional data, often leading to inaccurate predictions due to reliance on basic statistical measures and fixed threshold values, which fail to effectively separate outliers from normal data points.
Innovation Solution
A computer-program product that employs a quantile-based outlier filtration algorithm and matrix decomposition to detect and remove outliers, transforming data into a refined outlier-reduced dataset that improves predictive modeling by excluding anomalous values and mitigating heteroskedasticity and skewness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If basic statistical measures (mean and standard deviation) are used for outlier detection, then the process is simple and fast, but the detection accuracy deteriorates on complex or high-dimensional data
Solution Approach 1:
The patent transforms the data from its original high-dimensional space into a new coordinate system using Principal Component Analysis, changing the parameters representation. This allows the application of simple statistical measures in the transformed space while maintaining the ability to detect complex outliers in the original data structure.
Solution Approach 2:
The patent combines multiple techniques (PCA transformation, statistical outlier detection, and feature reconstruction) into a composite approach. This composite method leverages the strengths of each individual technique to achieve both computational efficiency and high detection accuracy on complex datasets.
2Ease of manufacture
If fixed threshold values are used in outlier detection algorithms, then the implementation is straightforward, but the ability to detect outliers in datasets with variable distributions deteriorates
Solution Approach 1:
The patent dynamically determines outlier thresholds based on the statistical properties of the transformed data rather than using fixed predetermined values. The thresholds are calculated adaptively from the data itself, allowing the algorithm to adjust to different data distributions while maintaining implementation simplicity.
3Reliability
If aggressive outlier removal is applied to eliminate all anomalies, then the dataset becomes cleaner, but important features associated with outliers may be lost
Solution Approach 1:
The patent carefully extracts only the outlier portions of the data in the transformed space while preserving the underlying feature structure. By working in the PCA-transformed space and selectively removing outliers based on their projection onto principal components, the method separates anomalies from important features rather than removing everything associated with outlier detection.
Data Source
AI summary
A computer-program product, computer-implemented method, and computer-implemented system includes obtaining a raw dataset; executing an outlier filtration process based on obtaining the raw dataset; training a model using a refined outlier-reduced dataset; and predicting, via the trained model, a value of the target entity at a future time.