ML Classifier Training Data Subset Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning classifiers face challenges in determining the optimal amount of training data required to achieve sufficient quality, as increasing data beyond a certain point often leads to diminishing returns, and existing methods lack efficiency in identifying this point, particularly concerning training time and resource utilization.
Innovation Solution
A method involving a computing device that selects subsets of training data, divides them into training and validation portions, trains a classifier, and determines precision, generating a graph to recommend the appropriate subset size for training, thereby optimizing data usage and reducing unnecessary computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the amount of training data is increased to improve model quality, then prediction accuracy is improved, but training time increases
Solution Approach 1:
The system performs preliminary experiments by training the classifier on progressively larger subsets of training data before final deployment. This preliminary action identifies the point of diminishing returns where additional data no longer significantly improves accuracy, allowing the selection of an optimal subset size that balances accuracy requirements with training time constraints
Solution Approach 2:
Instead of using the complete training dataset, the system uses partial subsets of varying sizes to train the classifier. By testing multiple subset sizes (e.g., 10%, 30%, 50%, 70%, 90% of total data), the system determines the minimum sufficient data required to achieve acceptable accuracy, avoiding the excessive use of full datasets
2Measurement precision
If the amount of training data is increased beyond a certain point, then model quality is marginally improved, but computational resource consumption increases significantly
Solution Approach 1:
The system performs preliminary resource consumption analysis by training classifiers on different subset sizes and measuring the associated computational costs. This preliminary action establishes the relationship between subset size and resource consumption, enabling the identification of the optimal point where marginal quality improvements no longer justify the increased resource expenditure
Solution Approach 2:
The system changes the parameter of training data subset size to optimize the balance between model quality and resource consumption. By systematically varying the subset size parameter and evaluating both quality metrics and resource usage, the system identifies the optimal configuration that achieves satisfactory performance with minimal resource expenditure
3Productivity
If a small subset of training data is used to reduce training time, then training efficiency is improved, but model quality may be insufficient
Solution Approach 1:
The system implements feedback loops where the classifier is trained on subsets of varying sizes and its performance is evaluated on validation data. This feedback mechanism provides quality metrics for each subset size, allowing the system to determine whether a given subset is sufficient or if larger subsets are needed to achieve the required quality threshold
Solution Approach 2:
The system uses partial subsets of training data with carefully selected sizes that are sufficient to achieve acceptable quality levels. Rather than using minimal subsets that might be insufficient, the system selects partial subsets that provide the necessary quality while maintaining training efficiency, avoiding both extremes of too little or excessive data
Data Source
AI summary
An embodiment may include a machine learning based classifier that maps input observations into respective categories and a database containing a corpus of training data for the classifier. The training data includes a plurality of entries, each entry having an observation respectively associated with a ground truth category thereof. A computing device may be configured to select, from the training data, a plurality of subsets each containing a different number of entries. The computing device may also be configured to, for each particular subset: (i) divide the particular subset into a training portion and a validation portion, (ii) train the classifier with the training portion, (iii) provide the validation portion as input to the classifier as trained, and (iv) based on how entries of the validation portion are mapped to the categories, determine a respective precision for the particular subset.