Cached Decoding for Static Inference Graphs in AI Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models, particularly auto-regressive transformer models, face inefficiencies due to duplicated computations and dynamic inference graph changes, leading to compatibility issues with AI accelerator frameworks that rely on static inference graphs.
Innovation Solution
Implementing cached decoding techniques using a key framer and value framer to cache key and value features across iterations, allowing the inference graph to remain static and avoiding duplicated computations by retrieving previously calculated features from a buffer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If feature tensors are recalculated in each iteration, then computation is straightforward and simple, but computational costs are high and efficiency is low
Solution Approach 1:
The patent applies preliminary action by pre-calculating and caching feature tensors (key and value features) from previous iterations before they are needed in subsequent iterations. The caching mechanism stores these pre-computed features in a buffer, so when a decoding iteration needs them, they are already available without requiring recalculation. This eliminates redundant computational work and significantly improves decoding efficiency while reducing computational costs.
2Adaptability or versatility
If the inference graph changes dynamically across iterations, then the model can adapt to sequential dependencies, but compatibility with AI accelerator frameworks is reduced
Solution Approach 1:
The patent applies segmentation by dividing the inference process into distinct static and dynamic components. The inference graph structure itself is kept static and compatible with AI accelerator frameworks, while the data flowing through it (specifically the cached feature tensors from previous iterations) carries the sequential dependency information. This segmentation allows the graph structure to remain framework-compatible while still handling sequential dependencies through the cached data inputs.
3Productivity
If feature tensors are stored in a buffer, then duplicated computations are avoided, but memory usage increases
Solution Approach 1:
The patent applies discarding and recovering by implementing a buffer with limited capacity that stores feature tensors from previous iterations. When the buffer becomes full, older entries are discarded to make room for new ones. This allows the system to recover (re-use) recently computed features without unbounded memory growth, balancing computation efficiency gains against memory consumption constraints.
Data Source
AI summary
Cached decoding systems and techniques are described. A system (e.g., decoder) receives an input token (e.g., input vector). The system applies a projection tensor (e.g., a projection matrix) to the input token to generate a feature tensor (e.g., a key tensor or a value tensor). The system processes at least the feature tensor and at least one previous feature tensor using at least one attention calculation to generate an output token. The at least one previous feature tensor is retrieved from a buffer. The at least one previous feature tensor can be stored in the buffer after having been previously calculated based on application of the projection tensor to a previous input token (e.g., from a previous iteration before the iteration in which the input token is received).