ML Model Selection via Runtime-Accuracy Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for selecting machine learning (ML) models based on training data are time-consuming and may prioritize accuracy over other important factors, leading to suboptimal results.
Innovation Solution
A method that calculates a final score for each ML model by evaluating training runtime using a predictive process, allowing for quicker selection of the most suitable model by incorporating both accuracy and runtime considerations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If ML models are ranked by accuracy values, then the selected model has optimal accuracy, but the selection process takes too much time
Solution Approach 1:
The patent applies preliminary action by training ML models on a subset of training data before final selection. This allows the system to evaluate model performance and runtime characteristics on smaller data portions first, then use these preliminary results to guide the final model selection process, reducing the overall time required while maintaining accuracy assessment
Solution Approach 2:
The patent implements partial action by using a subset of the complete training data for initial model evaluation. Instead of requiring full training data for the selection process, the system evaluates models on partial data to obtain runtime estimates and accuracy metrics, then selects the optimal model without needing to complete full training on all data for every model
2Measurement precision
If ML models are ranked by accuracy values, then the selected model has optimal accuracy, but other important factors are overlooked
Solution Approach 1:
The patent applies parameter changes by shifting from a single accuracy-based ranking metric to a multi-parameter evaluation system. The system calculates both accuracy values and runtime values, then combines these parameters to determine final model rankings. This allows the selection process to consider multiple factors including computational efficiency, resource requirements, and performance metrics simultaneously
Solution Approach 2:
The patent introduces another dimension by adding runtime evaluation as a separate evaluation axis alongside accuracy. Instead of ranking models on a single accuracy dimension, the system evaluates models in a two-dimensional space of accuracy versus runtime, enabling selection of models that optimize the balance between these competing factors rather than maximizing accuracy alone
3Measurement precision
If complete training data is used for evaluating all ML models, then accurate performance assessment is achieved, but the process becomes time-consuming
Solution Approach 1:
The patent applies preliminary action by performing initial model evaluations on subsets of training data before final selection. This preliminary evaluation provides sufficient information to compare models and identify candidates for final selection, avoiding the need to train and evaluate all models on the complete training data set
Solution Approach 2:
The patent implements segmentation by dividing the training data into subsets for different evaluation stages. The system evaluates models on smaller data segments first to obtain preliminary performance and runtime metrics, then uses these segmented evaluation results to guide the final model selection, rather than processing the complete data set for every model
Data Source
AI summary
A system may be configured to perform operations to select a machine learning model. The operations may include training machine learning models with training data of a training data set and obtaining a first value representing a first required runtime for training each machine learning model. The operations may include evaluating, based on the first value, a second value representing a second required runtime for training the machine learning model with a complete training data set. The operations may include calculating a final score for each machine learning model in a group of machine learning models, wherein the calculating is performed on a basis of the second values for the machine learning models, ranking the machine learning models based on the final score to obtain ranks, and selecting the machine learning model that has obtained a highest rank in the ranking.