Large-Model Inference Memory Swapping for Lower First-Word Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models experience increased first-word latency and inefficient GPU resource utilization due to high demand for video memory during inference processes, particularly in scenarios with large input sequences and multiple parallel processing requests, leading to queuing and prolonged computation times.
Innovation Solution
Implement an inference computation method that requests video memory space for partial computation layers of a large model, dumps inference results to CPU memory, and transmits them back to GPU memory when needed, reducing the overall demand for video memory and enabling parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If video memory space is requested for all computation layers simultaneously, then inference computation can be performed, but video memory demand increases and queuing time increases
Solution Approach 1:
The patent divides the N computation layers into multiple groups, where each group contains a subset of computation layers. Video memory space is requested and allocated for each group separately rather than all layers simultaneously. This segmentation reduces the peak video memory demand at any given time, allowing multiple processing requests to be handled concurrently and reducing queuing time, thereby improving inference throughput while maintaining computation accuracy.
2Productivity
If video memory space is requested for all computation layers, then complete inference computation is enabled, but GPU resource utilization becomes inefficient due to memory constraints
Solution Approach 1:
The patent implements dynamic video memory management by allocating memory for computation layers in batches rather than all at once. The system dynamically requests video memory space for each group of computation layers when needed, processes them, and releases the memory. This dynamic approach optimizes GPU resource utilization by matching memory allocation to actual computation needs, preventing memory bottlenecks while maintaining manageable system complexity through structured memory management.
3Reliability
If input prompt template length increases, then model processing capability is utilized, but inference time becomes longer and first-word latency increases
Solution Approach 1:
The patent performs preliminary processing by dividing computation layers into groups and pre-allocation of video memory space for each group before actual inference computation. This allows the system to prepare memory resources in advance for handling long input sequences, reducing the time penalty associated with processing lengthy prompt templates while maintaining the model's full processing capability and accuracy.
Data Source
AI summary
Embodiments of this specification provide inference computation methods and apparatuses for large models. In a method, a large model includes N computation layers. When performing inference computation, a graphics processing unit (GPU) requests first video memory space for partial computation layers of the N computation layers, performs inference computation on the partial computation layers, and stores obtained inference result data in the first video memory space. Then the inference result data are dumped from the first video memory space to CPU memory, where the released video memory space can be provided for other computation layers to perform inference computation. When the inference result data are needed, the inference result data in the CPU memory are transmitted to GPU video memory. When data processed by the large model include privacy data, privacy protection needs to be implemented for the data during the processing.