Machine Learning Training Data Selection for Distribution Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data pruning techniques for improving model robustness in machine learning are unreliable and fail to effectively align training and test distributions, leading to suboptimal performance in unseen data contexts.
Innovation Solution
A method for training a machine learning model by selecting a subset of training data based on minimizing dissimilarity between source and target data using maximum mean discrepancy, employing a radial basis function kernel, and optimizing a binary quadratic program to determine the selection vector.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing data pruning techniques are used to reduce training data size, then memory requirements and annotation costs are reduced, but model robustness and reliability deteriorate
Solution Approach 1:
The patent changes the selection criterion parameter from traditional methods (outlier removal, random sampling) to maximum mean discrepancy (MMD) based selection. This parameter change enables the selection of training samples that are most similar to the test distribution, thereby maintaining model robustness while reducing training data size. The MMD metric provides a quantitative measure of distribution similarity that guides the pruning process.
Solution Approach 2:
The patent applies local quality by selecting specific training samples based on their individual similarity to the test distribution rather than applying uniform pruning criteria. Each training sample is evaluated using MMD against the test distribution, and only samples with high similarity (low dissimilarity) are retained. This localized selection ensures that the reduced training set maintains the critical characteristics needed for robust model performance.
2Loss of information
If traditional data pruning criteria are applied to filter training data, then annotation requirements are reduced, but the reliability of producing robust models deteriorates
Solution Approach 1:
The patent changes the selection criterion from traditional annotation-based pruning to MMD-based distribution similarity measurement. This parameter change allows the method to identify and retain training samples that are most representative of the test distribution without relying on manual annotation or traditional heuristics, thereby maintaining model robustness while reducing annotation requirements.
Solution Approach 2:
The patent enables the training data selection process to be self-service by using the test distribution itself as the reference for selecting training samples. The MMD computation automatically identifies training samples that are similar to the test distribution without requiring external expert judgment or manual curation, thus reducing annotation requirements while maintaining reliability.
3Productivity
If a subset of source data is selected to reduce training complexity, then training time and computational resources are reduced, but model performance on unseen data deteriorates
Solution Approach 1:
The patent changes the selection parameter from random or heuristic-based sampling to MMD-based distribution similarity measurement. This parameter change ensures that the selected training subset maintains high similarity to the test distribution, thereby preserving model performance on unseen data while achieving reduced training complexity and improved training efficiency.
Solution Approach 2:
The patent performs preliminary action by pre-computing the MMD between training samples and the test distribution before the actual training process. This preliminary selection of high-similarity samples ensures that the subsequent training process is both efficient and effective, as the model is trained on data that is already optimized for generalization to unseen test data.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
There is described a computer implemented method of training a machine learning model. The method comprises obtaining source data and target data, selecting a subset of the source data based on minimising dissimilarity between the source data and the target data, and training the machine learning model using the selected subset of the source data.