Softmax Accelerator Using LUT Offsets for Quantized AI Processors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing processors in electronic devices face challenges in efficiently processing the softmax function due to its high computational complexity, leading to increased time and resource consumption, especially when a separate processor is used for this operation.
Innovation Solution
A processor is equipped with an accelerator that includes an input module, a distribution information processing module, and a look-up table (LUT) module to process the softmax function by generating input data distribution information, identifying maximum values, determining offset values, and using the LUT to output the softmax function values without performing full exponential and division operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If a separate processor is used to process the softmax function, then the processing capability for softmax is improved, but the time and resources are consumed due to data transmission
Solution Approach 1:
The patent merges the softmax processing capability directly into the neural network processor by integrating a lookup table module that stores pre-computed softmax values. This eliminates the need for a separate processor and avoids data transmission overhead, while maintaining the enhanced processing capability through dedicated hardware acceleration within the same processing unit.
Solution Approach 2:
The patent applies preliminary action by pre-computing softmax values for all possible quantized input combinations and storing them in a lookup table during processor initialization or fabrication. This allows the processor to retrieve pre-computed values during runtime without performing complex exponential and division operations, significantly reducing processing time and resource consumption.
2Measurement precision
If full exponential and division operations are performed for softmax, then the precision of the result is improved, but the computational complexity and time required are increased
Solution Approach 1:
The patent pre-computes softmax values for all possible quantized input combinations (e.g., 8-bit inputs resulting in 256x256=65,536 possible pairs) and stores them in a lookup table. During runtime, the processor simply retrieves pre-computed values based on input indices, eliminating complex exponential and division operations while maintaining precision through the use of sufficient bit-depth in the lookup table storage.
Solution Approach 2:
The patent creates a copy of all possible softmax result values in advance and stores them in the lookup table. Instead of computing results dynamically, the system copies the appropriate pre-computed value from the table based on input indices, significantly reducing computational complexity while preserving accuracy through high-precision storage of the copied values.
3Speed
If quantized input values are used, then the processing speed is improved, but the range of representable values is reduced
Solution Approach 1:
The patent changes the parameter of input representation from high-precision floating-point or high-bit integers to lower-bit quantized values (e.g., 8-bit quantization). This enables faster processing through simplified arithmetic operations and efficient lookup table indexing, while the system compensates for the reduced value range by applying offset adjustments and using the lookup table to map quantized inputs to accurate softmax outputs.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method performed by at least one processor configured to implement an accelerator for processing a softmax function includes: obtaining input data comprising a plurality of quantized input values; generating input data distribution information indicating a plurality of frequencies corresponding to the plurality of quantized input values included in the input data; identifying a largest value from among the plurality of quantized input values as a first maximum value, based on the input data distribution information; determining an offset value based on a difference between the first maximum value and a second maximum value, wherein the second maximum value indicates a maximum quantization value that is representable by an input value of the input data; determining a plurality of index values by applying the offset value to each quantized input value of the plurality of quantized input values; and outputting a value of the softmax function corresponding to each of the plurality of quantized input values based on the plurality of index values.