Neural Network Output Normalization via Approximated Softmax Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training artificial neural networks for classification tasks faces challenges due to the computational expense of softmax functions, especially on inference hardware, which can lead to unnormalized output ranges, making it difficult to compare scores across pixels or bounding boxes effectively.
Innovation Solution
Introducing a second loss function that normalizes the output of the artificial neural network by approximating the softmax function, allowing for the omission of exponential terms and ensuring the output values sum to 1, thereby enabling effective comparison of class scores across pixels or bounding boxes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the softmax function is used to normalize the output of the neural network, then the output values are normalized and can be compared effectively, but the computational cost increases due to exponential terms
Solution Approach 1:
The patent extracts and removes the exponential computation from the softmax function by using an alternative normalization approach. Instead of computing exp(x_i)/sum(exp(x_j)), the system uses a simplified normalization that achieves the same effect without the expensive exponential operations, thereby eliminating the harmful computational burden while preserving the normalization benefit
Solution Approach 2:
The patent replaces the computationally expensive softmax function with a cheaper alternative normalization method that can be computed efficiently. This substitute approach provides the necessary normalization effect at a fraction of the computational cost, making it suitable for resource-constrained inference hardware
2Loss of energy
If the softmax function is omitted to reduce computational cost, then the exponential terms are avoided, but the output values become unnormalized and difficult to compare
Solution Approach 1:
The patent introduces an intermediary normalization step that bridges the gap between raw neural network outputs and properly normalized probabilities. This intermediary process applies a simplified normalization algorithm that produces comparable output values without requiring the full softmax computation, thus maintaining measurement precision while reducing computational cost
3Ease of operation
If the neural network output is normalized during training, then the scores can be compared across pixels or bounding boxes, but additional loss function computation is required
Solution Approach 1:
The patent merges the normalization requirement into the existing training pipeline by using the same simplified normalization approach during both training and inference. This unified approach ensures score comparability across different inputs while avoiding the need for separate or complex normalization procedures, thereby easing operation without significantly increasing device complexity
Data Source
AI summary
A method is for training an artificial neural network, for classifying sensor data, as a function of a first loss function and a second loss function. The first loss function is calculated as a function of an output of the artificial neural network. The second loss function is configured such that the output of the artificial neural network is essentially normalized.