Large-Model Inference Memory Offloading for Lower First-Token Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models experience increased first-word latency and inefficient resource utilization due to high demand for video memory during inference processes, particularly in scenarios with long input sequences and multiple parallel processing requests, leading to queuing and poor throughput.
Innovation Solution
Implement an inference computation method that requests video memory space for partial computation layers, dumps inference results to CPU memory, and transmits them back to GPU memory as needed, reducing the overall memory demand and enabling parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If video memory space is requested for all computation layers to perform inference computation, then inference computation can be performed, but video memory demand increases leading to queuing and increased first-word latency
Solution Approach 1:
The computation layers are divided into multiple groups, with each group assigned to different GPU devices. This segmentation allows parallel processing of different layer groups across multiple GPUs, reducing the time required to allocate and use video memory for all layers, thereby decreasing first-word latency while maintaining inference throughput.
Solution Approach 2:
The patent introduces a dimensional change by distributing computation layers across multiple GPU devices rather than concentrating all layers on a single GPU. This multi-device dimensionality enables concurrent inference computation on different GPU devices, reducing video memory contention and first-word latency while improving overall system throughput.
2Reliability
If video memory space is requested for all computation layers, then complete inference computation can be performed, but resource utilization becomes inefficient due to high video memory demand
Solution Approach 1:
Computation layers are segmented and distributed across multiple GPU devices, allowing the system to maintain complete inference computation capability while improving resource utilization. Each GPU handles a subset of layers, preventing any single device from being overwhelmed and improving overall system efficiency.
Solution Approach 2:
The patent merges multiple GPU devices into a coordinated inference system, where each GPU contributes to the overall computation. This combining of resources allows the system to handle complete inference computations while distributing the memory burden, thereby improving resource utilization efficiency.
3Productivity
If video memory space is requested for multiple parallel processing requests, then more requests can be processed, but video memory demand increases causing queuing
Solution Approach 1:
Processing requests are segmented and assigned to different GPU devices based on the distributed layer architecture. This allows multiple requests to be processed in parallel across different devices without competing for the same video memory resources, reducing waiting time while maintaining high parallel processing capacity.
Solution Approach 2:
The patent adds a spatial dimension to parallel processing by distributing requests across multiple GPU devices. This multi-device dimensionality enables true concurrent processing of multiple requests, eliminating queuing delays that occur when requests compete for limited memory on a single device.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Embodiments of this specification provide inference computation methods and apparatuses for large models. In a method, a large model includes N computation layers. When performing inference computation, a graphics processing unit (GPU) requests first video memory space for partial computation layers of the N computation layers, performs inference computation on the partial computation layers, and stores obtained inference result data in the first video memory space. Then the inference result data are dumped from the first video memory space to CPU memory, where the released video memory space can be provided for other computation layers to perform inference computation. When the inference result data are needed, the inference result data in the CPU memory are transmitted to CPU video memory. When data processed by the large model include privacy data, privacy protection needs to be implemented for the data during the processing.