Softmax Prediction Circuit for Skipping Low-Probability Outputs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high complexity of non-linear functions, particularly softmax functions, in transformer models leads to increased latency and energy consumption, making efficient hardware implementation difficult.
Innovation Solution
A softmax low-probability output prediction circuit that predicts and skips unnecessary softmax calculations by using a shifter, subtractor, comparator, and controller to identify low-probability outputs, reducing complexity and energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If softmax function is applied to all elements in transformer attention mechanism, then calculation accuracy is maintained, but calculation complexity and energy consumption increase significantly
Solution Approach 1:
The patent applies partial action by performing softmax calculation only on selected elements rather than all elements. The prediction circuit identifies elements with high probability of low softmax output, and the controller selectively applies softmax only to elements that may produce significant attention weights, skipping elements predicted to have negligible contributions.
Solution Approach 2:
The patent extracts and processes only the necessary subset of elements through the prediction circuit. By predicting which elements will produce meaningful softmax outputs, the system separates important calculations from unnecessary ones, applying softmax only to the extracted relevant elements.
2Measurement precision
If softmax function is applied to all elements in transformer attention mechanism, then attention distribution accuracy is maintained, but latency increases
Solution Approach 1:
The patent implements preliminary action through the prediction circuit that evaluates elements before applying softmax. The prediction circuit performs preliminary assessment using arithmetic operations (addition, multiplication, comparison) to determine which elements warrant full softmax processing, filtering out elements that would produce negligible attention weights regardless of softmax application.
Solution Approach 2:
The patent applies skipping by allowing the controller to bypass softmax calculation for elements predicted to have low probability outputs. The system rushes through the evaluation phase using simple arithmetic operations and then skips the computationally expensive softmax function for identified low-probability elements, significantly reducing overall latency.
3Reliability
If softmax function is applied to all elements in transformer attention mechanism, then attention mechanism performance is maintained, but energy consumption increases
Solution Approach 1:
The patent applies partial action by performing softmax calculation only on selected elements rather than all elements. The prediction circuit identifies elements with high probability of low softmax output, and the controller selectively applies softmax only to elements that may produce significant attention weights, skipping elements predicted to have negligible contributions.
Solution Approach 2:
The patent changes the operational parameters by introducing a prediction-based filtering mechanism. The system dynamically determines which elements require full processing based on predicted probability thresholds, changing the active computation parameters from 'all elements' to 'selected elements based on prediction criteria.'
Data Source
AI summary
Proposed are a circuit and method for predicting a softmax low-probability output, and a softmax calculator. The circuit may include a first-in first-out (FIFO) memory configured to store all elements of a quantized input vector, and an accumulator configured to cumulatively add all the elements. The circuit may also include a shifter configured to calculate an arithmetic mean of all the elements by performing a right shift on a cumulative sum of all the elements. The circuit may further include a subtractor configured to calculate a result of subtracting the arithmetic mean from a specific one of all the elements, and a comparator configured to compare the subtraction result with a specific constant.