LLM Speculative Decoding with Paged KV Cache Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional large language models (LLMs) suffer from slow inference processes and low system throughput due to sequential dependency in token generation and inefficient use of graphics memory, leading to wasted memory space and limited batch processing capacity.
Innovation Solution
Implement a text generation method that combines speculative decoding and paging management in LLMs, where logical blocks are mapped to physical blocks based on a first criterion, allowing for increased token generation and batch storage in the key-value cache, thereby accelerating inference and improving hardware resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If speculative decoding is used to generate multiple candidate sequences, then token generation speed increases, but graphics memory usage increases
Solution Approach 1:
The patent divides the key-value cache into multiple logical blocks that can be independently managed and mapped to physical memory blocks. This segmentation allows efficient memory utilization when storing multiple candidate sequences from speculative decoding, as each sequence can be allocated specific logical blocks without requiring contiguous memory space, thereby reducing overall graphics memory usage while maintaining high token generation speed
Solution Approach 2:
The patent introduces a logical-physical block mapping dimension to separate logical memory addressing from physical memory allocation. By creating logical blocks that can be flexibly mapped to physical blocks, the system can store multiple candidate sequences efficiently in graphics memory, resolving the contradiction between generating multiple sequences (speed improvement) and memory consumption (resource constraint)
2Productivity
If multiple candidate sequences are stored in key-value cache, then system throughput increases, but memory space is wasted
Solution Approach 1:
The patent makes logical blocks universal by allowing them to serve multiple purposes: storing key-value pairs for different candidate sequences, being dynamically allocated and deallocated, and being mapped to different physical blocks as needed. This multi-functionality enables the key-value cache to efficiently accommodate multiple candidate sequences for parallel processing (increasing throughput) while avoiding memory waste through flexible resource utilization
Solution Approach 2:
The patent dynamically changes memory allocation parameters by adjusting the mapping between logical and physical blocks based on actual usage patterns. When candidate sequences are generated and validated, their memory blocks are allocated; when sequences are pruned or completed, blocks are deallocated and returned to the pool. This dynamic parameter adjustment ensures high system throughput while minimizing memory space waste
3Volume of stationary object
If logical blocks are mapped to physical blocks with deduplication, then memory efficiency improves, but mapping complexity increases
Solution Approach 1:
The patent performs preliminary actions by pre-allocating logical blocks and establishing the logical-physical mapping framework before speculative decoding generates candidate sequences. The mapping structure, including block allocation tables and mapping relationships, is prepared in advance, which simplifies the actual mapping process during inference and reduces the perceived complexity despite the sophisticated deduplication mechanism
Data Source
AI summary
This specification provides text generation methods, apparatuses, and storage medium devices. One method includes the following operations. In an iteration of a plurality of iterations under a large language model (LLM): estimating a first text sequence following a current text sequence based on a speculative decoding method, forming a plurality of candidate sequences based on the current text sequence and subsequences of the first text sequence, allocating logical blocks to text units in the plurality of candidate sequences in a key-value cache, to store attention information of the text units, mapping the allocated logical blocks to physical blocks based on a first criterion, and determining, by the LLM, a newly generated text unit in the iteration by using attention information of each candidate sequence in the key-value cache, to form a current text sequence for a next iteration of the plurality of iterations.