Heterogeneous LLM Prompt Attention Processing with Split KV-Caches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Large Language Model (LLM) systems face challenges in efficiently processing prompts due to memory architectures that are not cost-effective and do not leverage parallel processing structures, particularly in attention-based systems.
Innovation Solution
Implementing a prompt attention-processing subsystem and a generation attention-processing subsystem using neural processing units (NPUs), graphic processing units (GPUs), and high-speed parallel multiply and accumulate parallel processing, with separate memory buses for prompt and generation KV-caches, allowing for parallel processing of tokens and efficient token generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single attention-processing subsystem is used for both prompt and generation, then device complexity is reduced, but processing time increases due to sequential operation requirements
Solution Approach 1:
The single attention-processing subsystem is segmented into two independent subsystems: a prompt attention-processing subsystem and a generation attention-processing subsystem. This segmentation allows prompt processing and generation to occur in parallel, reducing total processing time while maintaining manageable system complexity through modular architecture.
2Device complexity
If prompt KV-cache and generation KV-cache share the same memory bus, then device complexity is reduced, but memory bandwidth efficiency decreases
Solution Approach 1:
The memory architecture is segmented into separate memory buses for prompt KV-cache and generation KV-cache. This segmentation enables independent memory access paths, allowing simultaneous memory operations without contention and maximizing memory bandwidth utilization for both processing stages.
3Measurement precision
If prompt processing is performed sequentially before generation, then processing accuracy is maintained, but productivity decreases due to sequential execution
Solution Approach 1:
The processing pipeline is segmented into independent prompt processing and generation processing stages that can execute in parallel. The prompt attention-processing subsystem prepares KV-caches while the generation attention-processing subsystem simultaneously generates output tokens, maintaining accuracy through proper data flow separation while doubling productivity through parallel execution.
Solution Approach 2:
The prompt attention-processing subsystem performs preliminary action by pre-computing and storing KV-caches in high-speed memory before generation begins. This preliminary preparation eliminates the need to recompute prompt representations during generation, enabling accurate parallel processing and improving throughput.
Data Source
AI summary
Methods and systems are disclosed for implementing a Large Language Model utilizing a prompt attention-processing subsystem and a generation attention-processing subsystem. A sequence of tokens is first processed by a prompt attention-processing subsystem, which utilizes an associated prompt KV-cache to store matrix values generated during prompt attention-processing. Upon the completion of prompt attention-processing, the populated prompt KV-cache is transferred to a generation KV-cache for processing by the generation attention-processing subsystem. The prompt and generation attention-processing subsystem can be multi-headed. The separate processing of the prompt facilitates efficient computations. Further, the prompt can be processed in segments that match available memory and computational resources. The generation attention-processing subsystem then produces an output token sequence based on the prompt KV-cache values transferred to the generation attention-processing system. The described system ensures optimized processor and memory usage and streamlined processing for large language model systems.