Non-Contiguous Attention Mask for Transformer KV Cache Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing transformer ANN models face inefficiencies in KV cache management due to memory limitations and static compiler frameworks, leading to high CPU load and latency in left-padded buffer implementations.
Innovation Solution
Implement a non-contiguous attention mask for KV cache management, allowing a right-padded buffer design that eliminates the need for memory movement and pointer shifts, maintaining performance without sacrificing precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a left-padded buffer implementation is used for KV cache management, then memory limitations can be addressed, but CPU load increases and latency increases
Solution Approach 1:
The patent inverts the conventional left-padded buffer approach by using a right-padded buffer implementation. Instead of padding with zeros on the left side of the KV cache buffer, the system pads on the right side, which fundamentally changes how the buffer is managed and eliminates the need for costly memory movement operations during inference.
Solution Approach 2:
The patent extracts and removes the unnecessary memory movement operations that are required in left-padded buffer implementations. By using a right-padded buffer with a non-contiguous attention mask, the system eliminates the need for shifting memory blocks or performing complex pointer manipulations, thereby reducing CPU load and improving inference speed.
2Quantity of substance
If a left-padded buffer implementation is used for KV cache management, then memory limitations can be addressed, but CPU load increases
Solution Approach 1:
The patent inverts the conventional left-padded buffer approach by using a right-padded buffer implementation. Instead of padding with zeros on the left side of the KV cache buffer, the system pads on the right side, which fundamentally changes how the buffer is managed and eliminates the need for costly memory movement operations during inference.
Solution Approach 2:
The patent extracts and removes the unnecessary memory movement operations that are required in left-padded buffer implementations. By using a right-padded buffer with a non-contiguous attention mask, the system eliminates the need for shifting memory blocks or performing complex pointer manipulations, thereby reducing CPU load and improving inference speed.
3Productivity
If a non-contiguous attention mask is used, then CPU load is reduced and inference speed increases, but memory layout becomes more complex
Solution Approach 1:
The patent introduces a non-contiguous attention mask as an intermediary structure that bridges the gap between the simplified right-padded buffer layout and the required token-specific KV vector access. This attention mask acts as a mediator that translates the simple memory layout into the correct token-specific selections without requiring complex memory manipulation.
Solution Approach 2:
The patent segments the KV cache buffer into distinct token-specific segments using the non-contiguous attention mask. Each token's relevant KV vectors are identified and separated through the mask mechanism, allowing independent processing while maintaining a simple overall memory layout. This segmentation enables efficient multi-stream token generation and draft model support.
Data Source
AI summary
A processor-implemented method includes constructing a non-contiguous attention mask corresponding to selected key-value (KV) vectors non-contiguously stored in a KV cache buffer. The method also includes multiplying the non-contiguous attention mask with the KV cache buffer to obtain token-specific KV vectors. The method further includes generating a new KV vector, with an artificial neural network transformer model during a current inference iteration, based on an input token and the token-specific KV vectors. The method may also append the new KV vector into an input buffer of the KV cache buffer adjacent to right-side padding during a next inference iteration with the artificial neural network transformer model.