Softmax Hardware Acceleration for Low-Latency Generative Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative AI models, particularly those based on transformer architecture, require significant computing resources due to frequent computations of the softmax function, leading to impractical scaling of hardware resources and operational latency.
Innovation Solution
A hardware accelerator module optimizes the computation of the softmax function by determining maximum and minimum input values for exponent multiplication, reducing the input value size, and using a lookup table to store and retrieve computed values, thereby reducing computational resources and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the softmax function is computed frequently for each inference iteration of the generative model, then the model can generate accurate and responsive output, but the operational latency and computational resources increase significantly
Solution Approach 1:
The patent pre-computes exponent values for a range of input values and stores them in a lookup table before the generative model inference begins. This preliminary action allows the softmax function to retrieve pre-computed values during inference instead of calculating exponents in real-time, significantly reducing operational latency while maintaining output accuracy.
2Productivity
If hardware resources are scaled up to accommodate the computational demands of the generative model, then the model can execute faster, but the hardware footprint and cost become impractical
Solution Approach 1:
The patent changes the computational parameters by replacing complex exponent multiplication operations with simple lookup table retrievals. This parameter change transforms the computational complexity from O(1) mathematical operations to O(1) memory access operations, achieving high execution speed with minimal hardware resources.
Solution Approach 2:
The patent creates a copy of exponent values in a lookup table that can be quickly accessed during inference. Instead of performing repeated exponent calculations, the system copies pre-computed values into a data structure that enables rapid retrieval, reducing the need for additional computational hardware.
3Quantity of substance
If quantization is applied to reduce data precision format, then the computational demand and model size are reduced, but further optimization through improved computational architecture is still needed
Solution Approach 1:
The patent substitutes the mechanical computation system (exponent multiplication operations) with a memory-based retrieval system (lookup table). This replacement eliminates the need for complex arithmetic operations during inference, significantly reducing computational demand and energy consumption while working complementarily with quantization techniques.
Data Source
AI summary
A computing system with hardware acceleration for execution of generative models is provided. The computing system comprises a processor and memory storing instructions that, when executed by the processor, cause the processor to execute a generative model. The computing system further comprises an accelerator module which performs compute operations during execution of the generative model. Prior to execution of the generative model, the accelerator module determines a maximum and minimum value for a functional computation to be performed during execution of the generative model. The accelerator module modifies possible inputs into functional computation to reduce the size of an input value by N bits. The accelerator module performs the functional computation based upon the modified input value, the minimum value, and the maximum value. During execution of the generative model, the accelerator module obtains a value for the functional computation to be used during generation of output of the generative model.