Transformer Generative Inference with Activation Sequence Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative models face challenges in real-time inference and decision-making due to high computational complexity and resource demands, necessitating methods that can perform inference with reduced computational resources.
Innovation Solution
Implementing a generative model with a transformer layer that includes a self-attention sub-layer and multilayer perception (MLP) sub-layer, utilizing an activation sequence caching technique to cache activations and reduce memory usage, thereby enhancing performance and minimizing processor communication overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If generative models are used for real-time inference, then decision-making capability is improved, but computational complexity and resource demands increase
Solution Approach 1:
The patent applies preliminary action by pre-computing and caching activation sequences from previous decoding stages. The caching mechanism stores intermediate computation results (activations from self-attention and MLP sub-layers) so that during subsequent inference steps, these pre-computed values can be reused directly rather than recalculating them, thereby reducing real-time computational complexity while maintaining inference capability
Solution Approach 2:
The patent implements dynamics by adaptively managing the caching mechanism based on computational needs. The system dynamically determines what activation sequences to cache and when to retrieve them from cache versus when to perform fresh computations, allowing the computational complexity to be adjusted according to the specific inference requirements and available resources
2Measurement precision
If generative models perform detailed computations, then inference accuracy is improved, but memory usage increases
Solution Approach 1:
The patent applies the taking out principle by extracting only the essential activation sequences from the full computational process and storing them in cache. Rather than caching all intermediate results or maintaining full computational states in memory, the system selectively caches only the activation sequences from self-attention and MLP sub-layers that are critical for maintaining inference accuracy, thereby reducing overall memory usage while preserving computational precision
Solution Approach 2:
The patent uses copying by creating cached copies of activation sequences that can be reused across multiple decoding stages. Instead of maintaining and manipulating the full original computation graphs in memory, the system creates compact cached copies of essential activation data, allowing accurate recomputations or retrievals with minimal memory footprint
3Speed
If activation sequences are cached in processor memory, then computational speed is improved, but communication overhead between processors increases
Solution Approach 1:
The patent applies the intermediary principle by introducing a caching mechanism as a mediator between processors. The cache serves as an intermediate storage layer that holds activation sequences, allowing processors to retrieve needed data locally from cache rather than constantly communicating with remote processors or main memory. This intermediary cache reduces communication overhead and energy loss while maintaining computational speed through fast local access
Data Source
AI summary
A method and an apparatus for inference using a generative model are provided. The method includes generating, by the one or more first processors executing the one or more transformer layers in a first decoding stage, a first output token by using a first input sequences based on a first input token, and generating, by the one or more first processors executing the one or more transformer layers in a second decoding stage, a second output token by using a second input sequence based on a second input token corresponding to the first output token.