Outlier Filtering via Graph Dimensionality Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data classification methods face challenges in efficiently identifying and filtering outliers from large datasets, particularly in image classification, where outliers can significantly affect the performance of recognition techniques and are costly to verify through crowdsourced labeling.
Innovation Solution
A method involving the construction of a graph data structure to reduce feature dimensionality using techniques like transductive diffusion component analysis (TDCA), followed by a pre-conditioned LASSO algorithm to identify and filter outliers, ensuring more robust classifiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated methods are used to collect and annotate images, then productivity is improved, but manufacturing precision deteriorates due to the presence of outliers in the annotated clusters
Solution Approach 1:
The patent applies preliminary action by performing outlier detection and removal before the classification training process. The system identifies and removes outlier images from the annotated clusters prior to training the classifier, ensuring that only high-quality inlier images are used for model training. This preliminary filtering step prevents outliers from degrading the classifier's performance while maintaining the efficiency of automated annotation.
Solution Approach 2:
The patent implements the taking out principle by extracting and removing outlier images from the annotated clusters. The system separates the data set into inliers (high-quality images) and outliers (low-quality images) based on outlier scores computed from graph-based feature dimensionality reduction. By extracting and removing these outliers, the system improves the precision of the training data while maintaining automated annotation efficiency.
2Measurement precision
If graph-based feature dimensionality reduction is applied, then measurement precision is improved for outlier detection, but device complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the outlier detection process into distinct stages: (1) constructing a graph data structure from the data set, (2) performing feature dimensionality reduction on the graph to obtain a reduced representation, and (3) computing outlier scores based on the reduced features. This segmentation allows the system to manage the complexity of graph-based methods while achieving high measurement precision in outlier detection.
Solution Approach 2:
The patent implements dimensionality change by transforming the original high-dimensional feature space into a lower-dimensional space through graph-based feature dimensionality reduction. The system constructs a graph where nodes represent data points and edges represent similarities, then reduces the feature dimensionality by analyzing the graph's topology and diffusion properties. This dimensionality reduction maintains the essential structure needed for accurate outlier detection while reducing computational complexity.
3Manufacturing precision
If pre-conditioned LASSO is used for outlier identification, then manufacturing precision is improved, but loss of information increases due to feature selection
Solution Approach 1:
The patent applies parameter changes by using pre-conditioned LASSO to selectively identify and remove outliers based on their computed scores, rather than removing data points based on arbitrary thresholds or losing entire feature dimensions. The LASSO algorithm applies L1 regularization to compute outlier scores that reflect the contribution of each sample to the overall data structure. This approach maintains precision in outlier identification while minimizing information loss by preserving all original features and only removing identified outliers from the training set.
Data Source
AI summary
Techniques are disclosed for identifying and filtering outliers from a sample set of data prior to training a classifier on an object using the sample set. A data set including a plurality of samples used to train a classification model is retrieved. The samples in the data set have a feature dimensionality. A graph of the data set is built. Each node in the graph corresponds to a sample in the data set and edges connecting the nodes correspond to a measure of similarity between the nodes. The feature dimensionality of the sample data set is reduced based on a topology of the graph. One or more outliers in the data set are identified based on the reduced feature dimensionality.