Learned Differentiable Loss Functions for Data-Aligned Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional loss functions for machine learning models are predefined and not based on the training data, leading to performance limitations and inefficiencies, such as increased processing cycles and misalignment with business objectives.
Innovation Solution
A learned loss function is generated by extracting values from a neural network's layers, training a surrogate model to output a symbolic equation, and applying this equation as a new loss function to improve model training, aligning it with the underlying data and business objectives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If predefined loss functions are used for training neural networks, then the training process is simple and straightforward, but the model performance is limited and does not align well with business objectives
Solution Approach 1:
The patent applies preliminary action by pre-training a neural network with predefined loss functions before learning the optimal loss function. This preliminary training phase prepares the model with initial weights and structure, which are then used as the foundation for the subsequent loss function learning phase, enabling the model to achieve better performance aligned with business objectives
Solution Approach 2:
The patent implements dynamics by transitioning from static predefined loss functions to a dynamic learned loss function. The loss function is no longer fixed but is instead learned and optimized during the training process based on the specific dataset and business objectives, allowing the training process to adapt and improve model performance
2Productivity
If predefined loss functions are used, then the training process is efficient, but processing cycles are increased and convergence is slower
Solution Approach 1:
The patent applies feedback by using the training process itself to generate information about optimal loss functions. The neural network's performance and the characteristics of the training data provide feedback that guides the learning of the loss function, enabling the system to automatically optimize training efficiency and reduce processing cycles through iterative improvement
3Measurement precision
If learned loss functions are generated and applied, then model accuracy and convergence are improved, but the training process becomes more complex
Solution Approach 1:
The patent uses preliminary action by first training the neural network with predefined loss functions to establish a baseline model. This preliminary phase simplifies the overall process by preparing the model structure and weights before the more complex loss function learning phase, making the transition to higher accuracy more manageable
4Adaptability or versatility
If conventional loss functions are used, then the training process is straightforward, but the model does not align well with business objectives
Solution Approach 1:
The patent implements dynamics by making the loss function adaptive rather than static. The learned loss function dynamically adjusts based on the specific business objectives and dataset characteristics, enabling the model to align better with business goals while the system automatically handles the complexity of adaptation
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Systems and methods are disclosed herein for training a model with a learned loss function. In an example system, a first trained neural network is generated based on application of a first loss function, such as a predefined loss function. A set of values is extracted from one or more of the layers of the neural network model, such as the weights of one of the layers. A separate machine learning model is trained using the set of values and a set of labels (e.g., ground truth annotations for a set of data). The machine learning model outputs a symbolic equation based on the training. The symbolic equation is applied to the first trained neural network to generate a second trained neural network. In this manner, a learned loss function can be generated and used to train a neural network, resulting in improved performance of the neural network.