ML Model Prefetching Across Memory Tiers for Lower Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face inefficiencies due to high access latency when fetching experts from slower memory tiers, leading to suboptimal bandwidth usage and increased execution times.
Innovation Solution
A prefetch module identifies and transfers a subset of experts from slower to faster memory tiers before model execution, using a prefetch table to optimize data movement and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If machine learning models fetch experts from slower memory tiers, then storage capacity is improved, but access latency increases
Solution Approach 1:
The system performs preliminary actions by identifying and fetching required expert models from slower storage to faster memory before they are actually needed for inference. The prefetch module proactively loads models into the buffer based on prediction requests, so that when inference is needed, the models are already available in fast memory, thus resolving the latency issue while maintaining access to large-scale storage.
2Loss of time
If machine learning models use faster memory tiers, then access latency is reduced, but bandwidth usage becomes suboptimal
Solution Approach 1:
The system applies partial action by loading only the specific expert models that are predicted to be needed into the fast memory buffer, rather than loading all possible models. This selective prefetching approach ensures that fast memory bandwidth is used efficiently for loading only relevant models, avoiding wasted bandwidth while still achieving low latency for the required models.
3Productivity
If machine learning models increase execution speed, then productivity is improved, but memory access efficiency decreases
Solution Approach 1:
The system introduces an intermediary component - the prefetch module with its buffer - that mediates between the slow storage and the computation units. This intermediary pre-loads models into fast memory based on predicted needs, allowing the computation units to execute at high speed without being bottlenecked by memory access delays, thus decoupling execution speed from memory access efficiency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods for fetching machine learning models are disclosed. A processor identifies an input to a first machine learning model having a first layer and a second layer. The processor identifies from a table, based on the input, a second machine learning model associated with the first layer and a third machine learning model associated with the second layer. Based on identifying the second machine learning model and the third machine learning model from the table, the processor transmits a command to fetch the second machine learning model and the third machine learning model from the first storage medium into the second storage medium. and executes the second machine learning model and the third machine learning model for generating a prediction based on the input.