Disaggregated Speculative Decoding for Low-Latency LLM Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing LLM-serving systems face challenges in achieving ultra-low latency decoding in single-request scenarios due to the inherent trade-off between throughput and latency, with speculative decoding methods being ineffective when combined with tensor parallelism, as the draft and target models are co-located, leading to inefficiencies in scaling and overheads.
Innovation Solution
The method involves redesigning the speculative decoding pipeline in an asynchronous and disaggregated manner, allocating the draft and target models to different processors for parallel computation, utilizing parallel tree generation, consistent KV cache management, and latency-optimized kernels to decouple dependencies and reduce overheads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If speculative decoding is combined with tensor parallelism with co-located draft and target models, then model verification can be performed, but scaling efficiency deteriorates and overhead increases
Solution Approach 1:
The system segments the speculative decoding process into distinct draft model and target model components that operate independently on separate processor devices. This segmentation allows each model to be optimized and scaled independently, improving overall scaling efficiency while maintaining verification accuracy through their coordinated interaction.
Solution Approach 2:
The patent introduces an intermediary communication mechanism between the draft model and target model that enables efficient token verification. The draft model generates candidate tokens that are then verified by the target model through an optimized verification process, reducing overhead while maintaining reliability.
2Loss of information
If draft and target models are co-located on the same processor, then communication overhead is reduced, but parallel computation efficiency deteriorates
Solution Approach 1:
The patent transitions from a single-processor co-located architecture to a multi-processor distributed architecture. By adding the dimension of spatial distribution across multiple processor devices, the system achieves both reduced communication overhead through optimized data transfer protocols and improved parallel computation efficiency through simultaneous draft and target model operations on different devices.
3Device complexity
If sequential decoding is used, then implementation simplicity is maintained, but decoding latency increases
Solution Approach 1:
The draft model performs preliminary action by generating candidate tokens in advance before the target model verifies them. This preliminary generation of multiple candidate tokens allows the system to explore multiple decoding paths simultaneously, significantly reducing decoding latency while maintaining implementation simplicity through the structured draft-verify workflow.
Solution Approach 2:
The system maintains continuity of useful action by having the draft model continuously generate candidate tokens while the target model continuously verifies them in a pipelined manner. This continuous parallel operation eliminates idle time and reduces overall decoding latency compared to sequential decoding, while the structured pipeline maintains implementation simplicity.
Data Source
AI summary
Embodiments of the present disclosure provide a method, device, and medium for improving latency. The method comprises receiving a plurality of verified tokens. And the method further comprises generating, by the first model and based on the plurality of verified tokens, a plurality of candidate tokens. And the method further comprises sending the plurality of candidate tokens to the second model, wherein the first model is allocated to at least one first processor, and the second model is allocated to at least one second processor, and the at least one first processor is used for computation of the first model, and the at least one second processor is used for computation of the second model respectively, and the computation of the second model is carried out in parallel during the computation of the first model.