AutoML Task Grouping for Benchmark Evaluation Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for evaluating the generalization performance of Automated Machine Learning (AutoML) are time-consuming and resource-intensive due to the large amount of benchmark data and task sets required.
Innovation Solution
A task control program is implemented on a computer-readable recording medium that executes AutoML processing on multiple tasks, classifies tasks into groups based on similarities of pipelines and their evaluation values, and selects tasks with the shortest execution times to generate a task group.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If AutoML processing is executed on a large amount of benchmark data and task sets to evaluate generalization performance, then the evaluation comprehensiveness is improved, but the evaluation time and resource consumption increase significantly
Solution Approach 1:
The patent segments the benchmark task set into multiple groups based on task categories and characteristics. By dividing the large-scale benchmark into smaller manageable groups, the system can evaluate generalization performance across different task segments without requiring exhaustive processing of all tasks simultaneously, thus reducing overall evaluation time while maintaining comprehensive coverage.
Solution Approach 2:
The patent performs preliminary classification and organization of benchmark tasks before conducting AutoML evaluation. Tasks are pre-grouped by similarity metrics and evaluation criteria are prepared in advance, allowing the evaluation process to proceed more efficiently without redundant computations, thereby reducing evaluation time while preserving evaluation comprehensiveness.
2Measurement precision
If diverse benchmark tasks are used to maintain task variety in evaluation, then the generalization performance assessment accuracy is improved, but the complexity of task management and evaluation coordination increases
Solution Approach 1:
The patent implements a universal task grouping framework that can handle multiple types of benchmark tasks (e.g., classification, regression, clustering) through a common evaluation architecture. This multi-functional approach allows diverse tasks to be managed under unified principles, maintaining assessment accuracy across different task types while reducing the complexity of individual task management.
Solution Approach 2:
The patent varies evaluation parameters such as grouping criteria, similarity thresholds, and task selection metrics based on the specific characteristics of different benchmark sets. By dynamically adjusting these parameters, the system maintains high assessment accuracy for diverse tasks without requiring complex fixed management structures for each task type.
Data Source
AI summary
A non-transitory computer-readable recording medium stores a task control program for causing a computer to execute a process including: executing automated machine learning (AutoML) processing on each of a plurality of tasks to acquire a plurality of pipelines for each of the plurality of tasks; classifying the plurality of tasks into a plurality of groups based on similarities of one or more pipelines selected based on evaluation values, among the plurality of pipelines, and similarities of evaluation values of the one or more pipelines; and generating a task group by selecting one task from each of the plurality of groups based on an execution time of the AutoML processing of each of the plurality of tasks.