Hyperparameter Selection via Aggregate Loss Minimization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The inefficiency and high computational cost associated with manually selecting hyperparameter values for machine learning tasks, particularly as models become more complex, necessitates a more efficient method for determining optimal hyperparameter settings.

Innovation Solution

A computer-implemented method that generates an ordered list of hyperparameter sets by evaluating candidate sets across multiple machine learning tasks to minimize aggregate loss, reducing the need for manual selection and resource-intensive quasi-random search algorithms.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If manual selection of hyperparameter values is used, then flexibility and control are maintained, but efficiency and productivity deteriorate significantly

Engineering Contradiction:

Improvehyperparameter selection controlVSAvoidmodel training efficiency

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The system performs self-service by automatically selecting optimal hyperparameter values through a learned model that evaluates candidate hyperparameters based on task characteristics, eliminating the need for manual selection while maintaining optimal performance

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system changes the approach from manual parameter selection to automated parameter selection by a learned model, transforming the hyperparameter selection process from a manual operation to an automated intelligent process

Inventive Principle:

Principle #35Parameter changes

2Extent of automation

If quasi-random search algorithms are used for hyperparameter selection, then automation is improved, but computational cost and resource usage worsen prohibitively

Engineering Contradiction:

Improvehyperparameter selection automationVSAvoidcomputational resource consumption

Core Design Contradiction:

Extent of automationVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary action by pre-selecting and ordering candidate hyperparameter sets based on task characteristics before actual model training begins, eliminating the need for resource-intensive random search during the training phase

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system substitutes the mechanical quasi-random search algorithm with an intelligent learned model that predicts optimal hyperparameters directly, replacing brute-force computational methods with intelligent prediction

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Adaptability or versatility

If more hyperparameters are included in complex models, then model capability and adaptability are improved, but the complexity of manual selection worsens

Engineering Contradiction:

Improvemodel capabilityVSAvoidhyperparameter selection complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system achieves universality by creating a general learned model that handles hyperparameter selection across multiple different machine learning tasks and model types, making the solution applicable to a wide range of scenarios rather than task-specific

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20230059708A1Generation of Optimized Hyperparameter Values for Application to Machine Learning Tasks

Publication Date: 2023.02.23 GOOGLE LLC

AI summary

The present disclosure provides a computer-implemented method for determining an optimized list of sets of hyperparameter values for application to an additional machine learning task. The method includes obtaining data describing a plurality of different machine learning tasks. The method includes obtaining a plurality of candidate sets of hyperparameter values. The method includes determining an ordered list of sets of hyperparameters selected from the plurality of candidate sets of hyperparameter values, wherein the ordered list of sets of hyperparameters minimizes an aggregate loss over the plurality of different machine learning tasks. The method includes storing the ordered list of sets of hyperparameters for use in training an additional machine learning model to perform an additional machine learning task.