Neural Network Loss Function Balancing Incorrect Category Probabilities
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network training methods, particularly for classification tasks, do not adequately account for the distribution of probabilities across incorrect categories, making networks susceptible to outputting incorrect answers due to minor perturbations.
Innovation Solution
Incorporating loss function terms that evenly distribute probabilities for incorrect categories while maximizing the probability for the correct category, using differentiable functions to approximate maximum and minimum values of incorrect category probabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If cross-entropy loss function is used that only considers correct category probability, then training simplicity is maintained, but network robustness deteriorates
Solution Approach 1:
The loss function is segmented into multiple independent terms: the standard cross-entropy term for correct category probability, and additional terms for handling incorrect category probabilities. This segmentation allows each term to address specific aspects of the problem independently while collectively improving robustness without complicating the overall training process
Solution Approach 2:
The patent introduces an intermediary mechanism (the distributional regularization terms) that mediates between the simple cross-entropy objective and the desired robustness. These intermediary terms gradually guide the network to distribute probabilities more evenly across incorrect categories, bridging the gap between simplicity and robustness
2Productivity
If probability distribution for incorrect categories is not considered, then computational efficiency is maintained, but susceptibility to perturbations increases
Solution Approach 1:
The patent modifies the loss function parameters by adding terms that specifically target the distribution of probabilities across incorrect categories. This parameter change transforms the loss landscape to encourage more robust probability distributions without significantly increasing computational complexity, as the additional terms can be computed efficiently during backpropagation
3Reliability
If probability distribution for incorrect categories is considered, then network robustness is improved, but loss function complexity increases
Solution Approach 1:
The patent applies partial action by focusing the additional loss terms specifically on the distribution among incorrect categories, rather than redesigning the entire loss function. This targeted approach improves robustness by addressing only the specific deficiency in how incorrect category probabilities are handled, without unnecessarily complicating other aspects of the training process
Data Source
AI summary
Some embodiments provide a method for training a machine-trained (MT) network to classify inputs into multiple categories. The method propagates a set of input training items through the MT network to generate a set of corresponding outputs. Each input training item belongs to a category and the output for each input training item includes, for each category, a computed probability of the input belonging to the category. The method computes a value for a loss function based on the generated outputs. The loss function includes a first term based on the computed probabilities of each input belonging to its category and not based on individual computed probabilities of the inputs belonging to other categories and a second term based on the individual computed probabilities of each input belonging to each of the categories. The method uses the computed value for the loss function to train the MT network.