Foundation Model Swapping Across Memory Layers for Faster Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI model swapping techniques are inadequate for foundation models (FMs), which require significant GPU memory and often exceed available memory limits, leading to inefficient use and high costs due to hosting unused models.
Innovation Solution
A method and apparatus for AI model swapping between processor, host, and disk memory layers, using a weighted eviction score based on least recently used time, model size, and future usage to optimize memory usage and reduce loading time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If multiple AI models are loaded into GPU memory simultaneously, then model availability and response time are improved, but memory consumption and cost increase significantly
Solution Approach 1:
The system dynamically loads and unloads AI models based on real-time demand and usage patterns. Instead of statically loading all models, the system activates models only when needed and swaps them out when no longer required, optimizing the balance between response time and memory consumption
Solution Approach 2:
The system changes the state of model availability by transitioning models between loaded and unloaded states based on demand. This parameter change allows the system to maintain fast response times for active models while reducing overall memory consumption by keeping inactive models in a swapped-out state
2Reliability
If foundation models are loaded into GPU memory, then inference capability is improved, but memory availability for other models decreases
Solution Approach 1:
The system segments the model storage into multiple memory layers: GPU memory for active foundation models, CPU memory for swapped-out models, and disk storage for archival. This segmentation allows foundation models to be loaded when needed while maintaining the ability to support multiple different models across the segmented memory hierarchy
Solution Approach 2:
The system introduces CPU memory as an intermediary layer between GPU memory and disk storage. This intermediary allows foundation models to be swapped between GPU and CPU memory, enabling the GPU to dedicate its limited memory to a single foundation model while the CPU memory serves as a buffer for other models, thus maintaining both inference capability and model variety support
3Productivity
If AI models are swapped between memory layers, then memory utilization is improved, but swapping overhead and complexity increase
Solution Approach 1:
The system performs preliminary actions by pre-loading models into CPU memory before they are needed in GPU memory. This preliminary placement in CPU memory reduces the complexity of direct GPU-disk swapping and allows for more manageable memory management operations
Solution Approach 2:
The system uses feedback from model usage patterns and performance metrics to dynamically adjust swapping decisions. By monitoring which models are actively used and when, the system optimizes swap timing and target selection, reducing unnecessary swapping operations and simplifying the overall management complexity
Data Source
AI summary
There is provided a method and apparatus for model selection for swapping between a first memory and a second memory. The method includes sorting models in a first list, the first list including models loaded in the first memory, wherein sorting is based at least in part on a least recently used (LRU) of each model. The method further includes removing duplicate models from a second list of models, the second list including models to be loaded into the first memory. The method further includes determining an eviction score of each of the models and selecting a model from the first list having a highest eviction score compared with other models in the first list. The method additionally includes swapping the selected model from the first memory to the second memory.