Noise-Level Segmentation for Machine Learning Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training machine learning models are inefficient due to excess noise in the data, leading to suboptimal model performance and high computational costs.
Innovation Solution
The method involves categorizing training data into subsets based on noise levels and training separate machine learning models for each subset, allowing for improved prediction performance without discarding valuable data or using extensive noise correction techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If data cleansing is used to fill unavailable features with default values, then the completeness of the dataset is improved, but the noise level in the training data increases
Solution Approach 1:
The patent segments the training data into multiple subsets based on noise levels. Each subset contains data points with similar noise characteristics, allowing the model to learn from heterogeneous data without being adversely affected by high noise in any single subset. This resolves the contradiction by preserving incomplete data (improving completeness) while isolating its noise impact through segmentation.
2Object-affected harmful factors
If all data points with unavailable features are removed, then the noise level in the dataset is reduced, but the size of the dataset significantly decreases
Solution Approach 1:
Instead of removing noisy data points, the patent segments them into appropriate noise-level subsets. This allows retention of all valuable data (maintaining dataset size) while organizing noisy and clean data separately so that each subset can be trained appropriately, resolving the contradiction between preserving data quantity and managing noise levels.
Solution Approach 2:
The patent applies different training strategies to different subsets of data based on their local noise characteristics. Low-noise subsets are treated differently from high-noise subsets, allowing the system to maximize utilization of clean data while still incorporating noisy data in a controlled manner, thus preserving dataset size without uniformly compromising quality.
3Object-affected harmful factors
If noise correction methods are applied to remove noise from the data, then the quality of the training data is improved, but the computational resources required increase significantly
Solution Approach 1:
The patent performs preliminary segmentation of data into noise-level subsets before training begins. This pre-organization avoids the need for computationally expensive noise correction methods during training, as the noise management is handled through the subset structure rather than active correction algorithms, thus improving data quality handling while minimizing computational resource usage.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
There is provided a computer implemented method for training and validating a machine learning model. The method comprises receiving a dataset (111) comprising multiple datapoints, wherein each datapoint has associated with it one or more features, determining a noise level of each datapoint of the dataset (111), and categorising each datapoint of the dataset (111) into one of a plurality of data subsets (113-123), said categorising such that each data subset (113-123) comprises datapoints having a specific range of noise level, wherein the specific range of noise level is different for each subset (113-123). The method further comprises training a plurality of machine learning models (125-135), each machine learning model of the plurality of machine learning models (125-135) is trained on a specific one of the data subsets (113-123), and validating an output of each of the trained plurality of machine learning models (125-135) to determine a performance of the trained machine learning models (125-135) through comparing the output to a predetermined value.