Outlier Detection in Machine Learning Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Detecting outliers in training data for machine learning systems is challenging, as outliers can lead to skewed predictions and are difficult to handle due to their potential significance.
Innovation Solution
A computer-implemented method that uses a machine learning system and multiple robust measures of scale formulas to select upper and lower bounds for outlier detection, allowing for the identification and modification of training data to improve model accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If outliers are removed from training data, then model accuracy is improved, but loss of information occurs
Solution Approach 1:
The patent extracts and removes only the harmful portions of outlier data (erroneous values) while preserving the useful information. By using robust statistical methods to identify and remove only the erroneous parts of outlier records rather than discarding entire records, the system achieves improved model accuracy while minimizing information loss.
Solution Approach 2:
The patent changes the parameter of data handling by transitioning from simple deletion to selective removal of erroneous portions. By modifying how outliers are processed - specifically by identifying and removing only the harmful components while retaining useful information - the system resolves the contradiction between accuracy improvement and information preservation.
2Reliability
If robust statistical methods are used for outlier detection, then detection reliability is improved, but device complexity increases
Solution Approach 1:
The patent segments the outlier detection process into distinct stages: data collection, robust statistical analysis, outlier identification, and selective removal. By dividing the complex task into manageable segments, the system achieves reliable detection while maintaining manageable complexity through structured processing steps.
Solution Approach 2:
The patent introduces robust statistical methods as intermediary tools between raw data and model training. These statistical methods act as mediators that filter and prepare the data, achieving reliable outlier detection while the systematic approach to implementation keeps the overall complexity manageable through standardized procedures.
3Productivity
If all training data is processed, then productivity is maintained, but harmful factors increase
Solution Approach 1:
The patent converts the harmful presence of outliers into a beneficial detection opportunity. By using robust statistical methods to identify outliers, the system transforms what would be harmful data into useful information about data quality issues, enabling selective removal of harmful factors while maintaining productivity through automated detection processes.
Solution Approach 2:
The patent implements feedback mechanisms where the robust statistical analysis continuously monitors training data and provides feedback about outlier presence. This feedback loop enables the system to automatically identify and remove harmful factors while maintaining high productivity through automated, continuous processing without manual intervention.
Data Source
AI summary
One or more computer processors receiving training data comprising a trial subset of training data. The one or more computer processors probe the trial subset of training data using a machine learning system and multiple robust measures of scale formulas to select an upper bound for data outlier detection and to select a lower bound for data outlier selection. The one or more computer processors detect one or more outliers in the training data using the selected upper bound and the selected lower bound. The one or more computer processors generate modified training data using the detected outliers. The one or more computer processors train the machine learning system utilizing the modified training data.