Generative Model Inference With Activation Sequence Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative models face challenges in real-time inference and decision-making due to high computational complexity and resource demands, necessitating methods that can perform inference with reduced computational resources.
Innovation Solution
Implementing a generative model with a transformer layer that includes a self-attention sub-layer and multilayer perception (MLP) sub-layer, utilizing an activation sequence caching technique to cache sequences of activations, thereby reducing memory usage and enhancing performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If generative models are used for real-time inference, then decision-making capability is improved, but computational complexity and resource demands increase
Solution Approach 1:
The patent applies preliminary action by pre-computing and caching activation sequences from the encoder and previous decoder steps before the actual inference step. This allows the model to reuse these pre-computed values instead of recalculating them, significantly reducing the computational complexity during real-time inference while maintaining the decision-making capability.
Solution Approach 2:
The patent segments the computational process into distinct components: encoder outputs, cross-attention computations, and MLP operations. By caching the activation sequences from each segment separately, the system reduces redundant computations in each segment during inference, thereby lowering overall computational complexity while preserving real-time performance.
2Quantity of substance
If activation sequences are cached, then memory usage is reduced, but computational overhead increases
Solution Approach 1:
The patent extracts and caches only the essential activation sequences that are reused across multiple inference steps, rather than caching all intermediate computations. This selective extraction reduces memory usage while minimizing the additional computational overhead required for cache management, as only critical values are stored and retrieved.
3Reliability
If transformer layers are executed with full input sequences, then inference accuracy is maintained, but resource consumption increases
Solution Approach 1:
The patent uses preliminary action by pre-computing encoder activation sequences and caching them before the decoder processing. This allows the decoder to reuse these sequences without re-computation, reducing resource consumption during inference while maintaining the accuracy required for proper sequence generation.
Solution Approach 2:
The patent segments the computation into encoder and decoder parts, caching only the encoder outputs that are reused by the decoder. This segmentation reduces resource consumption by avoiding redundant computation of encoder sequences for each decoder step, while preserving the full input sequence information needed for accurate inference.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and an apparatus for inference using a generative model are provided. The method includes generating, by the one or more first processors executing the one or more transformer layers in a first decoding stage, a first output token by using a first input sequences based on a first input token, and generating, by the one or more first processors executing the one or more transformer layers in a second decoding stage, a second output token by using a second input sequence based on a second input token corresponding to the first output token.