Programmable Smart NICs for LLM Key-Value Cache Offload
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The large memory requirements of key-value (KV) caches in GPU accelerators for large language models (LLMs) lead to memory congestion, while transferring these caches to remote storage nodes introduces significant network overhead and latency due to CPU involvement in data transfer.
Innovation Solution
Utilizing programmable smart NICs to offload data transfer of KV caches between compute and storage nodes through triggered operations, reducing CPU involvement and optimizing data transfer with RDMA and prefetch mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If KV cache is stored in GPU memory, then fast access during LLM inference is achieved, but memory congestion occurs due to large memory requirements
Solution Approach 1:
The KV cache is segmented into two parts: a local cache in GPU memory for frequently accessed recent tokens, and a remote cache in CPU memory or storage for historical tokens. This segmentation allows the system to maintain fast access for active computation while offloading less frequently accessed data to free up GPU memory resources.
Solution Approach 2:
The system transitions from a single-memory-location architecture to a distributed memory architecture across multiple dimensions (GPU memory hierarchy and system memory). By adding the dimension of remote storage accessible via PCIe, the system resolves the contradiction between local fast access and overall memory capacity.
2Quantity of substance
If KV cache is transferred to remote storage nodes, then GPU memory is freed up, but significant network overhead and latency are introduced due to CPU involvement in data transfer
Solution Approach 1:
The system performs preliminary actions by pre-fetching KV cache data from remote storage into GPU memory before it is needed for computation. This allows the GPU to access the data with minimal latency when required, while still maintaining the benefit of having freed up GPU memory for other purposes during the prefetch operation.
Solution Approach 2:
The PCIe bus serves as an intermediary between GPU memory and system memory, enabling direct data transfer without requiring CPU intervention for each transfer operation. This intermediary mechanism reduces the overhead and latency associated with CPU involvement in data transfer.
3Adaptability or versatility
If CPU is involved in data transfer operations, then data management flexibility is maintained, but transfer overhead and latency increase
Solution Approach 1:
The GPU is empowered to perform self-service by directly managing its own cache data through PCIe access to system memory. The GPU can initiate and complete data transfer operations independently without CPU intervention, thereby maintaining data management flexibility while significantly improving transfer efficiency and reducing latency.
Solution Approach 2:
The CPU is extracted from the data transfer path, removing it as a bottleneck in the transfer operation. The system design allows GPU and system memory to communicate directly via PCIe, eliminating the need for CPU involvement in routine cache management while preserving CPU availability for higher-level data management decisions.
Data Source
AI summary
A first NIC monitors a key-value cache associated with an LLM executed by a compute node that includes the first NIC and an accelerator. The key-value cache is stored in a memory associated with the accelerator. Responsive to detecting that the key-value cache is updated by the accelerator, the first NIC transfers a copy of the key-value cache update to a remote storage node. The key-value cache is deleted from the memory after the query is inferred. Responsive to receiving a follow-up user query, the first NIC determines a storage location on the remote storage node that stores the key-value cache corresponding to the user query and sends a KV-cache-transfer request to a second NIC on the remote storage node, the KV-cache-transfer request specifying the storage location, thereby facilitating the second NIC to transfer the key-value cache corresponding to the user query from the specified storage location to the memory.