AI Data Processing Error Detection and Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models trained with incomplete or poor-quality data sets produce inaccurate results, and there is a need for techniques to improve the accuracy of these models and the relevance of input variable data sets, especially in large and diverse data environments.
Innovation Solution
A computer-implemented method that involves loading a training data set with multiple bins, applying under-sampling and over-sampling techniques, merging the updated bins, and training the model, while also generating and filtering input variables to emphasize their relevance, and automatically determining optimal hyperparameters for the model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models are trained with incomplete or poor-quality training data sets, then the training process can proceed with available data, but the accuracy of the results deteriorates
Solution Approach 1:
The system performs preliminary actions by detecting missing data ranges before training, generating synthetic data to fill these gaps, and preprocessing input variables to ensure quality. This preparatory work ensures that the training data set is complete and high-quality before the actual training process begins, thereby improving model accuracy without requiring additional real-world data collection
Solution Approach 2:
The system creates copies of existing data by generating synthetic data samples that replicate the characteristics of available data within important ranges. This synthetic data copying technique allows the model to learn from sufficient examples even when real data is incomplete, maintaining training accuracy without needing more original data
2Reliability
If automated techniques are applied to improve the relevance of input variable data sets, then the quality of input variables improves, but the processing time and complexity increase
Solution Approach 1:
The system performs self-service by automatically detecting the relevance of input variables and performing frequency filtering without manual intervention. The automated identification of important ranges and automatic filtering of input variables based on frequency analysis enables the system to improve data relevance while minimizing the need for manual processing, thereby reducing overall processing time despite the enhanced analysis
3Reliability
If under-sampling and over-sampling techniques are applied to balance training data bins, then the model accuracy improves, but the data processing complexity increases
Solution Approach 1:
The system segments the training data into distinct bins representing different classes or categories, then applies targeted under-sampling and over-sampling techniques to each bin independently. This segmentation approach allows for precise control over data balance while maintaining manageable processing complexity, as each bin can be processed separately with standardized sampling operations
Solution Approach 2:
The system changes the parameters of data sampling by dynamically adjusting sampling rates for different bins based on their characteristics. By modifying sampling parameters adaptively rather than applying uniform processing, the system achieves balanced data distribution and improved model accuracy while keeping processing complexity reasonable through parameter optimization
Data Source
AI summary
A computer-implemented method includes loading a training data set including a first bin and a second bin. The method includes applying an under-sampling technique to elements of the first bin to generate an updated first bin, applying an over-sampling technique to elements of the second bin to generate an updated second bin, generating an updated training data set by merging the updated first bin and the updated second bin, loading baseline hyperparameters, configuring a machine learning model with the baseline hyperparameters, providing the updated training data set to the configured machine learning model to determine baseline performance metrics, determining whether the baseline performance metrics are above a threshold, saving the baseline hyperparameters as optimal hyperparameters in response to determining that the baseline performance metrics are above the threshold, configuring the machine learning model with optimal hyperparameters, and providing input variables to the configured machine learning model to generate output variables.