LLM KV-Cache Offloading via Smart NICs to Free GPU Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) require significant memory for their key-value (KV) cache, which can lead to memory constraints on GPUs and increased latency due to data transfers between GPUs and remote storage nodes, involving CPUs and network stacks.
Innovation Solution
The use of programmable or smart Network Interface Controllers (NICs) on compute and storage nodes to offload data transfers from accelerators, such as GPUs, through triggered operations, allowing the KV cache to be transferred to remote storage nodes without CPU involvement, thereby reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the KV cache is stored in GPU memory, then the LLM inference can proceed efficiently with fast access to intermediate representations, but the GPU memory becomes constrained and cannot be used for other applications
Solution Approach 1:
The KV cache storage is segmented into two parts: a small portion kept in GPU memory for fast access during active inference, and the majority stored in CPU memory or remote storage. This segmentation allows the system to maintain fast inference performance while freeing up GPU memory for other computations.
Solution Approach 2:
The CPU acts as an intermediary between the GPU and remote storage for KV cache management. The CPU handles the transfer of KV cache data between GPU memory and CPU/remote storage, allowing the GPU to focus on computation while the CPU manages memory allocation and data transfer operations.
2Quantity of substance
If the KV cache is transferred to remote storage nodes, then GPU memory is freed for other applications, but data transfer latency increases due to CPU and network stack involvement
Solution Approach 1:
The system performs preliminary actions by pre-transferring KV cache data to CPU memory before it is needed, and by pre-warming the remote storage with anticipated KV cache contents. This allows the GPU to retrieve data faster when needed, reducing the perceived latency impact.
Solution Approach 2:
The CPU continuously manages KV cache transfers between GPU, CPU memory, and remote storage, maintaining an optimized distribution of cache data. This continuous management ensures that frequently accessed KV cache entries remain in fast memory while less frequently accessed data is stored remotely, minimizing transfer latency for active inference operations.
3Adaptability or versatility
If the CPU is involved in KV cache transfer operations, then the transfer can be managed flexibly, but the CPU becomes a bottleneck and GPU idle time increases
Solution Approach 1:
The system extracts the KV cache management function from the GPU and assigns it to the CPU and smart NIC. This extraction allows the GPU to focus entirely on computation while the CPU and smart NIC handle memory management and data transfer operations, preventing GPU idle time and maintaining high utilization.
Solution Approach 2:
The smart NIC provides self-service capabilities by autonomously managing KV cache transfers between GPU, CPU memory, and remote storage without requiring continuous CPU intervention. This self-service approach reduces CPU bottleneck effects and allows the GPU to maintain high productivity while the smart NIC independently handles cache management operations.
Data Source
AI summary
A first NIC monitors a key-value cache associated with an LLM executed by a compute node that includes the first NIC and an accelerator. The key-value cache is stored in a memory associated with the accelerator. Responsive to detecting that the key-value cache is updated by the accelerator, the first NIC transfers a copy of the key-value cache update to a remote storage node. The key-value cache is deleted from the memory after the query is inferred. Responsive to receiving a follow-up user query, the first NIC determines a storage location on the remote storage node that stores the key-value cache corresponding to the user query and sends a KV-cache-transfer request to a second NIC on the remote storage node, the KV-cache-transfer request specifying the storage location, thereby facilitating the second NIC to transfer the key-value cache corresponding to the user query from the specified storage location to the memory.