Early-Exit DNN Inference With Pipelined Constant Batch Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing early-exit Deep Neural Networks (DNN) models face inefficiencies due to batch size shrinkage during inference, leading to underutilization of GPUs and poor throughput, despite their potential for optimal latency and accuracy tradeoffs.
Innovation Solution
The method involves splitting the early-exit DNN model into smaller pieces and executing them in a model-parallel, pipelined fashion on heterogeneous resources, maintaining a constant batch size using online batch profile estimation and dynamic programming optimization to optimize split locations and resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Duration of action of moving object
If early-exit DNN models are used to achieve optimal latency and accuracy tradeoffs, then inference latency is reduced, but batch size shrinks during inference leading to GPU underutilization and poor throughput
Solution Approach 1:
The DNN model is segmented into multiple independent stages, where each stage can process batches of data. By dividing the model into stages 1 through N, the system can maintain larger batch sizes across stages while allowing individual samples to exit at different stages based on their complexity, thus resolving the conflict between low latency and high throughput
Solution Approach 2:
The system dynamically determines the exit stage for each sample based on its complexity and confidence level. This dynamic approach allows the batch size to be maintained at each stage while individual samples can leave early when their inference requirements are met, optimizing both latency and throughput simultaneously
2Duration of action of moving object
If early-exit DNN models are used to reduce inference time, then latency is improved, but resource utilization deteriorates due to batch size shrinkage
Solution Approach 1:
By segmenting the model into multiple stages with independent resource allocation, the system can maintain consistent batch sizes across stages, ensuring efficient GPU utilization while still allowing early exits for simple samples
Solution Approach 2:
The pipelined execution ensures that GPU resources continue to process data continuously across multiple stages. While some samples exit early, other samples continue processing in parallel, maintaining continuous useful action and high resource utilization without idle GPU cycles
3Measurement precision
If model size is increased to improve accuracy, then model performance is enhanced, but inference cost increases due to stricter SLO requirements and larger resource requirements
Solution Approach 1:
The large accurate model is segmented into multiple stages, where simpler samples can exit at earlier stages with lower computational cost, while only complex samples require the full model capacity. This reduces the average inference cost while maintaining high accuracy for difficult cases
Solution Approach 2:
Different stages of the model have different computational complexities and accuracy characteristics. The system applies local quality by allowing simple samples to be processed with lower computational resources at early stages, while reserving full model capacity for complex samples that require higher accuracy
Data Source
AI summary
The present disclosure relates to methods and systems for providing inferences using machine learning systems. The methods and systems receive a load forecast for processing requests by a machine learning model and split the machine learning model into a plurality machine learning model portions based on the load forecast. The methods and systems determine a batch size for the requests for the machine learning model portions. The methods and systems use one or more available resources to execute the plurality of machine learning model portions to process the requests and generate inferences for the requests.