ML Expert Prefetching for Low-Latency Memory Fetch
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face inefficiencies due to high access latency when fetching experts from slower memory to faster memory, leading to suboptimal bandwidth usage and increased execution latency.
Innovation Solution
A prefetch module identifies and retrieves a subset of experts from a slower memory to a faster memory before model execution, using a prefetch table populated with expert information per token and layer, optimizing data movement and bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models are fetched from slower memory to faster memory during execution, then model availability is improved, but access latency increases and bandwidth usage decreases
Solution Approach 1:
The system performs preliminary actions by identifying and fetching required machine learning model layers (experts) from slower memory to faster memory before they are actually needed for inference. The prefetch module uses the input token and layer information to proactively retrieve the necessary model components, so that when the inference process reaches those layers, the models are already available in fast memory, thereby eliminating access latency during critical execution paths.
2Reliability
If machine learning models are fetched from slower memory to faster memory during execution, then model availability is improved, but bandwidth usage deteriorates
Solution Approach 1:
The system performs preliminary actions by identifying and fetching required machine learning model layers (experts) from slower memory to faster memory before they are actually needed for inference. The prefetch module uses the input token and layer information to proactively retrieve the necessary model components, so that when the inference process reaches those layers, the models are already available in fast memory, thereby eliminating access latency during critical execution paths.
Solution Approach 2:
The system changes the timing parameter of memory access operations by shifting the fetch operation from synchronous (during execution) to asynchronous (before execution). By changing when the data transfer occurs, the system optimizes bandwidth utilization by performing transfers during idle periods or in parallel with other operations, rather than competing for bandwidth during the critical inference execution path.
3Reliability
If machine learning models are fetched during execution, then model availability is improved, but execution latency increases
Solution Approach 1:
The system performs preliminary actions by identifying and fetching required machine learning model layers (experts) from slower memory to faster memory before they are actually needed for inference. The prefetch module uses the input token and layer information to proactively retrieve the necessary model components, so that when the inference process reaches those layers, the models are already available in fast memory, thereby eliminating access latency during critical execution paths.
Data Source
AI summary
Systems and methods for fetching machine learning models are disclosed. A processor identifies an input to a first machine learning model having a first layer and a second layer. The processor identifies from a table, based on the input, a second machine learning model associated with the first layer and a third machine learning model associated with the second layer. Based on identifying the second machine learning model and the third machine learning model from the table, the processor transmits a command to fetch the second machine learning model and the third machine learning model from the first storage medium into the second storage medium. and executes the second machine learning model and the third machine learning model for generating a prediction based on the input.