Outlier Removal for Sparse Machine Learning Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for predicting physical properties using machine learning struggle with decreased accuracy due to erroneous data or outliers in training data, especially when the data is sparse, making it difficult to properly remove outliers.
Innovation Solution
An outlier removal method and device that calculate prediction errors by repeatedly dividing training data into teaching and test data, creating a regression model, and determining outliers based on index values characterizing the distribution of prediction errors, allowing for the removal of outliers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional outlier removal methods are used on sparse training data, then the data processing is simple and fast, but the outlier detection accuracy is poor and unreliable
Solution Approach 1:
The training data is divided into multiple subsets through repeated random partitioning into teaching data and test data. This segmentation allows the system to evaluate each data point's prediction error across multiple different contexts, improving outlier detection accuracy by considering varied scenarios rather than relying on a single data split.
Solution Approach 2:
The system performs preliminary regression model creation and prediction error calculation for each data point before making the final outlier determination. By pre-calculating prediction errors across multiple random partitions and storing these results, the system prepares comprehensive error distributions in advance, enabling more accurate outlier identification without repeating computationally intensive operations during the final determination phase.
2Reliability
If more training data is collected to improve machine learning accuracy, then the prediction model becomes more robust, but the difficulty of identifying outliers increases due to data sparsity in each individual partition
Solution Approach 1:
The system performs repeated periodic cycles of random data partitioning, regression model creation, and prediction error calculation. By executing this process multiple times with different random partitions, the system accumulates comprehensive prediction error distributions for each data point, making outlier detection more reliable even when individual partitions are sparse.
Solution Approach 2:
The system uses prediction errors from multiple random partitions as feedback to build comprehensive error distributions for each data point. These error distributions serve as feedback information that helps distinguish true outliers from normal variations, improving the reliability of outlier identification by comparing each data point's error pattern against the accumulated feedback from multiple partitions.
3Measurement precision
If repeated cross-validation with multiple regression models is performed to improve outlier detection, then the detection accuracy improves, but the computational time and processing complexity increase
Solution Approach 1:
The system performs preliminary calculation of prediction errors for each data point across multiple random partitions and stores these results in advance. By pre-computing and storing the error distributions before final outlier determination, the system avoids repeating computationally intensive regression model creation and prediction calculations during the final outlier identification phase, thus reducing overall computational time while maintaining detection accuracy.
Data Source
AI summary
An outlier removal method for removing an outlier included in training data that has data of an explanatory variable and an objective variable used for machine learning. The method includes calculating prediction errors by repeating, a predetermined number of times, division of the training data into teaching data and test data, creation of a regression model representing a correlation between the explanatory variable and the objective variable using the teaching data, and calculation of a prediction error using the test data on the created regression model, calculating a distribution by extracting, for each data included in the training data, prediction errors when using that data as the test data from prediction errors obtained in the calculating prediction errors, and obtaining, for each data included in the training data, an index value characterizing a distribution of the extracted prediction errors, determining an outlier by determining whether each data is an outlier based on the index value of each data obtained in the calculating a distribution, and removing an outlier by removing data determined to be an outlier in the determining an outlier.