Passive and Active Outlier-Tolerant Learning With Test Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning methods lack effective solutions for identifying and tolerating outliers in training data, leading to poor passive and active outlier tolerance, resulting in regression and clustering deviations due to the presence of abnormal samples.
Innovation Solution
A passive and active outlier-tolerant learning method that involves acquiring a train and test set, judging the distribution of abnormal samples, applying corresponding outlier-tolerant algorithms for feature learning, training an initial model, processing the test set with an outlier-correction algorithm, and testing the intermediate model to obtain a final learning model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional machine learning methods are used without outlier handling, then the learning process is simple, but the reliability of learning results deteriorates due to outlier influence
Solution Approach 1:
The patent applies preliminary action by performing outlier detection and handling before the main learning process. The method first identifies abnormal samples in the training set, then processes them through imputation or removal operations before proceeding with model training. This preliminary processing ensures that the subsequent learning process works with cleaned data, improving reliability without adding complexity to the core learning algorithm.
Solution Approach 2:
The patent introduces an intermediary component - the outlier detection and processing module - that acts as a mediator between the raw training data and the learning algorithm. This intermediary identifies and handles abnormal samples through statistical methods or machine learning-based outlier detection, then feeds processed data to the learning model, thereby protecting the learning process from outlier interference while maintaining a clear separation of concerns.
2Manufacturing precision
If data cleaning is performed to remove abnormal samples, then the quality of training data improves, but the loss of useful information increases due to potential removal of valid samples
Solution Approach 1:
The patent applies parameter changes by using statistical parameters (such as mean, standard deviation, or distribution characteristics) to identify outliers rather than using fixed thresholds. The method dynamically adjusts outlier detection parameters based on the data distribution, allowing it to distinguish between genuine outliers and valid but extreme samples. This parameter-based approach maintains data quality while minimizing information loss by only removing truly abnormal samples.
Solution Approach 2:
The patent incorporates feedback mechanisms where the outlier detection process iteratively refines its identification of abnormal samples. The system detects outliers, processes them, evaluates the impact on learning results, and adjusts the outlier handling strategy accordingly. This feedback loop ensures that useful information is preserved while effectively removing harmful outliers, as the system learns from the consequences of its outlier handling decisions.
3Reliability
If outlier-tolerant algorithms are applied to handle abnormal samples, then the passive outlier tolerance improves, but the computational cost increases due to additional processing steps
Solution Approach 1:
The patent applies segmentation by dividing the outlier handling process into distinct modular stages: outlier detection, outlier classification (determining whether to remove or impute), and outlier processing. Each stage can be independently optimized and executed only when necessary. This segmented approach improves passive outlier tolerance through systematic processing while controlling computational cost by avoiding redundant operations in each stage.
4Reliability
If comprehensive outlier detection and processing is implemented, then the active outlier tolerance improves, but the complexity of the system increases due to multiple processing stages
Solution Approach 1:
The patent applies universality by designing a multi-functional outlier handling system that can perform multiple operations (detection, removal, imputation, transformation) through a unified framework. The same core infrastructure supports different outlier processing strategies, allowing the system to adapt to various data types and outlier patterns without requiring separate complex systems for each function. This universal approach improves active outlier tolerance while managing system complexity through code reuse and standardized interfaces.
Data Source
AI summary
The present disclosure relates to a passive outlier-tolerant and active outlier-tolerant learning method with abnormal samples and a system thereof. The method includes the following steps: (1) judging the distribution of abnormal samples in a train set, and calling a corresponding outlier-tolerant algorithm based on the distribution to carry out feature learning on the sample feature quantity in the train set to obtain an outlier-tolerant learning result; (2) training an initial learning model by using the outlier-tolerant learning result to obtain an intermediate learning model; (3) processing the test set by a preset outlier-correction algorithm to obtain an outlier-corrected test set; (4) testing the intermediate learning model using the outlier-corrected test set to obtain a final learning model. The scheme of the present disclosure can improve the passive outlier tolerance and active outlier tolerance of a machine learning process to abnormal samples.