Neural Network Confidence Calibration via Adaptive Label Smoothing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image recognition networks using deep learning suffer from overconfident predictions due to the use of softmax cross entropy loss, which results in inaccurate confidence values and reduced reliability in decision-making.
Innovation Solution
An apparatus and method for calibrating confidence using adaptive conditional label smoothing, which involves determining correct and false logit elements, calculating calibration values based on their differences, and applying these values to the target probability distribution to prevent overconfident predictions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If softmax cross entropy loss is used to train the network, then the network can learn to make class-wise estimated probability follow target probability distribution, but the confidence value becomes too large compared to accuracy (overconfident prediction)
Solution Approach 1:
The patent changes the parameter of target probability distribution from one-hot labels (0 or 1) to smoothed labels (0 < q < 1). This parameter change prevents the network from overfitting to extreme probability values, thereby calibrating confidence values to be more accurate and reliable. The smoothed target probability distribution ensures that the network learns to output confidence values that better reflect actual prediction accuracy.
2Loss of energy
If the correct class estimate probability is learned to reach 1 to match the one-hot label target probability, then the network minimizes cross entropy loss, but the network overfits and makes overconfident predictions
Solution Approach 1:
The patent applies beforehand cushioning by using smoothed target probability distributions (0 < q < 1) instead of extreme one-hot labels. This cushioning effect prevents the network from overfitting to perfect probability values during training, thereby preventing overconfident predictions in the future. The smoothed labels act as a regularizer that balances loss minimization with prediction reliability.
3Ease of operation
If the network outputs high confidence values to indicate high probability predictions, then decision-making becomes straightforward, but the confidence values do not match actual accuracy
Solution Approach 1:
The patent implements feedback through the use of smoothed target probability distributions that provide continuous guidance during training. The smoothed labels (0 < q < 1) create a feedback mechanism that adjusts the network's confidence outputs to better match actual accuracy. This feedback loop ensures that high confidence values are only assigned when the network is truly confident, improving both decision-making ease and confidence accuracy.
Data Source
AI summary
An apparatus for calibrating confidence determines a correct logit element and a false logit element among a plurality of logit elements of a logit vector obtained by a neural network model performing a neural network operation on the training data, based on a correct class and a false class determined from a target probability distribution including target probabilities for each of a plurality of classes according to ground truth of the training data as elements, and determines whether to calibrate and a calibration value for a plurality of target probabilities of the target probability distribution based on a difference in value between the correct logit element and the false logit element.