Pipeline Model Selection via Iterative Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current model selection methods in computing systems are inefficient in automatically selecting models that balance accuracy and performance, often leading to long selection times and models that fail to meet user requirements.
Innovation Solution
A computer-implemented method for model selection that involves building models to predict pipeline model characteristics, receiving user specifications, defining a metric for weighing model characteristics, iteratively reducing the number of pipeline models, determining the best model capable of meeting user specifications, and deploying that model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple pipeline models are evaluated to ensure accurate model selection, then model selection accuracy is improved, but selection time increases
Solution Approach 1:
The system performs preliminary evaluation of pipeline models by executing them on sample data before final selection. This preliminary action filters out inadequate models early in the process, allowing the system to maintain high selection accuracy while reducing the time required for evaluating all candidate models exhaustively.
Solution Approach 2:
The model evaluation process is divided into multiple phases: initial filtering phase where models are quickly assessed on sample data, and a final selection phase where only promising candidates undergo comprehensive evaluation. This segmentation allows the system to balance accuracy requirements with time constraints by applying different evaluation depths at different stages.
2Reliability
If comprehensive model evaluation is performed to meet user specifications, then model reliability is improved, but computational resources are consumed
Solution Approach 1:
The system performs partial evaluation of models by testing them on representative sample data rather than exhaustive datasets. This partial action is sufficient to filter out models that cannot meet user specifications, thereby maintaining reliability while significantly reducing computational resource consumption during the selection process.
Solution Approach 2:
Comprehensive model evaluation is performed only as a preliminary filtering step before deployment. Models that pass this preliminary reliability check are then deployed without requiring continuous exhaustive verification, thus ensuring model reliability while conserving computational resources during operation.
3Productivity
If simple model selection criteria are used, then selection speed is improved, but model performance may not meet user requirements
Solution Approach 1:
The selection criteria are segmented into two levels: simple criteria (execution time on sample data, basic accuracy metrics) used for rapid initial filtering, and comprehensive criteria (detailed performance evaluation, user specification matching) applied only to models that pass the initial filter. This segmentation maintains selection speed while ensuring model performance requirements are met.
Solution Approach 2:
Simple evaluation criteria are applied as a preliminary filtering mechanism to quickly identify promising models. This preliminary action maintains high selection speed while ensuring that models meeting user requirements are not overlooked, as they would pass the initial simple criteria check before more thorough evaluation.
Data Source
AI summary
A computer-implemented method for model selection is provided. The computer-implemented method includes building models for predicting characteristics of pipeline models, receiving user specifications for pipeline model performance, defining a metric for weighing the characteristics of the pipeline models, using the metric to iteratively reduce a number of the pipeline models capable of meeting the user specifications to a reduced number of the pipeline models, determining which one pipeline model of the reduced number of the pipeline models exhibits a best capability of meeting the user specifications and deploying the one pipeline model.