Machine Learning Training with Non-Differentiable Hyperparameter Gradients
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning training methods for computer-based vision require additional validation data sets and are limited by the use of differentiable hyperparameters, which can be inefficient and costly.
Innovation Solution
A method that initializes both model parameters and metaparameters, allowing for efficient calculation of gradients for non-differentiable hyperparameters without the need for validation data, thereby enhancing generalization and convergence of the training process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If gradient-based approaches are used for hyperparameter adaptation, then model parameter gradients become more advantageous, but validation data sets become necessary which increases data requirements and costs
Solution Approach 1:
The patent extracts the validation function into the training process itself by using a subset of training data to compute gradients for hyperparameter optimization. This eliminates the need for separate validation datasets while maintaining the ability to guide hyperparameter adaptation through gradient-based methods.
Solution Approach 2:
The training data serves multiple functions simultaneously: it is used both for computing model parameter gradients and for computing hyperparameter gradients. This multi-functionality eliminates the need for dedicated validation data while maintaining effective hyperparameter adaptation.
2Productivity
If traditional hyperparameter tuning methods are used, then training can proceed, but non-differentiable hyperparameters cannot be efficiently optimized
Solution Approach 1:
The patent transforms the hyperparameter optimization problem into a differentiable form by computing gradients with respect to hyperparameters using training data. This allows continuous and differentiable hyperparameters to be optimized efficiently through gradient-based methods, improving training speed while maintaining adaptability.
3Measurement precision
If validation data are created to improve training accuracy, then model performance improves, but costs increase due to the effort required to create validation datasets
Solution Approach 1:
The system uses its own training data to perform the validation function automatically. By computing hyperparameter gradients on training data, the system eliminates the need for externally created validation datasets, reducing data creation costs while maintaining training accuracy through gradient-based hyperparameter adaptation.
Data Source
AI summary
A computer-implemented method for training a machine learning system including: initializing parameters of the machine learning system and a metaparameter. Repeatedly carrying out the following as a loop: providing a batch of training data points and manipulating the provided training data points or a training method for optimizing the parameters of the machine learning system or a structure of the machine learning system based on the metaparameter. Ascertaining a cost function as a function of instantaneous parameters of the machine learning system and of the instantaneous metaparameters. Adapting the instantaneous parameters as a function of an ascertained first gradient, which has been ascertained with respect to the instantaneous parameters via the ascertained cost function for the training data points, and adapting the metaparameter as a function of a second gradient, which has been ascertained with respect to the metaparameter used in the preceding step via the ascertained cost function.