Softmax Output Layer Pre-computation for Neural Network Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network output processing methods are inefficient due to high computational complexity, especially when using quantized arithmetic, which limits the range of potential values and requires extensive exponentiation operations.
Innovation Solution
The system determines the number of occurrences of each possible output value in the softmax layer output and calculates exponentiation measures and normalization factors pre-computationally, allowing for efficient normalization without the need for hardware or software support for exponentiation operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If quantized arithmetic is used in neural network processing, then the range of potential values is limited, but computational complexity increases due to extensive exponentiation operations
Solution Approach 1:
The patent applies preliminary action by pre-computing the softmax output values and storing them in a lookup table before runtime. During actual neural network inference, the system simply retrieves pre-computed values based on quantized input indices, avoiding real-time exponentiation operations. This resolves the contradiction by preparing computational results in advance, reducing runtime complexity while maintaining full value range capability.
2Measurement precision
If standard softmax computation is performed without pre-computation, then accurate probability values are obtained, but extensive exponentiation operations increase processing time
Solution Approach 1:
The system pre-computes softmax probability values for all possible quantized input combinations and stores them in a lookup table. During inference, it retrieves pre-computed accurate probability values instantly by indexing with quantized inputs, eliminating runtime exponentiation while preserving numerical accuracy. This resolves the time-accuracy tradeoff by performing computationally intensive operations beforehand.
Solution Approach 2:
The patent creates a copy of softmax computation results in the form of a lookup table containing pre-computed probability values. Instead of重新 computing exponentiation operations during inference, the system copies and retrieves results from the stored table, significantly reducing processing time while maintaining accuracy. This copying approach resolves the contradiction between computation time and result accuracy.
3Measurement precision
If exponentiation operations are performed in real-time, then accurate softmax output is generated, but hardware or software support for exponentiation is required, increasing system complexity
Solution Approach 1:
The system pre-computes softmax outputs including all exponentiation operations during a setup phase, storing results in a lookup table. During actual use, only simple table indexing and retrieval operations are needed, eliminating real-time exponentiation requirements. This resolves the contradiction by moving complex computational requirements to a preliminary setup phase, simplifying runtime hardware/software requirements while preserving output accuracy.
Solution Approach 2:
The patent copies the results of complex exponentiation-based softmax computations into a lookup table structure. This copying transforms the computational problem from requiring active exponentiation capabilities to requiring only memory access operations, reducing hardware/software complexity while maintaining numerical accuracy through the pre-computed copied values.
Data Source
AI summary
Systems, methods, and apparatus, including computer programs encoded on a computer storage medium for processing a network input through a neural network having one or more initial neural network layers followed by a softmax output layer. In one aspect, the methods include obtaining a layer output generated by the one or more initial neural network layers and processing the layer output through the softmax output layer to generate a neural network output. Processing the layer output through the softmax output layer includes determining, for each possible output value, a number of occurrences in the layer output values; for each possible output value occurring in the layer output values, determining a respective exponentiation measure; determining a normalization factor for the layer output by combining the exponentiation measures in accordance with the number of occurrences of the possible output values; and determining, for each of layer output values, a softmax probability value.