Neural Network Inference Batching With Adaptive Pipelining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Batch processing in deep neural networks is resource-constrained, and the batch size affects the execution code, necessitating optimal batch size determination and pipelining decisions to enhance resource utilization and inference efficiency.
Innovation Solution
A method and apparatus that predict the input data number for subsequent intervals using neural network-based models, determine batch sizes, and adjust pipelining based on resource states to optimize neural network inference by dynamically adapting the batch size and pipelining strategy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch size is increased to improve resource utilization, then productivity increases, but device complexity and resource constraints worsen
Solution Approach 1:
The patent implements dynamic batch size adjustment by predicting input data numbers for subsequent intervals and adapting batch sizes in real-time based on predicted data arrivals and resource states. This resolves the contradiction by making the batch size flexible rather than fixed, allowing the system to optimize resource utilization without permanently overloading the accelerator.
Solution Approach 2:
The system uses feedback mechanisms by monitoring resource states (utilization, queue depth, latency) and using this information to adjust batch sizes and pipelining decisions. The feedback loop enables the system to respond to actual resource constraints while maintaining high productivity through adaptive optimization.
2Productivity
If batch size is increased to improve throughput, then productivity increases, but inference latency increases
Solution Approach 1:
The patent implements periodic batch processing with dynamic batch size adjustment based on time intervals and predicted data arrivals. By organizing processing into periodic batches with adaptive sizing, the system achieves high throughput while controlling latency through timely batch formation and execution.
Solution Approach 2:
The system dynamically adjusts batch sizes based on real-time conditions including predicted input data numbers and resource states. This dynamic adaptation allows the system to optimize the trade-off between throughput (larger batches) and latency (smaller, faster batches) based on current operational conditions.
3Productivity
If pipelining is enabled to improve processing efficiency, then productivity increases, but device complexity increases
Solution Approach 1:
The patent implements preliminary action by determining pipelining decisions in advance based on predicted resource states and input data patterns. The system pre-configures pipelining parameters before execution, reducing runtime complexity while maintaining processing efficiency through proactive optimization.
4Adaptability or versatility
If dynamic batch size adjustment is implemented to optimize resource utilization, then adaptability increases, but device complexity increases
Solution Approach 1:
The system implements self-service by using automated neural network-based prediction models to determine optimal batch sizes without manual intervention. The prediction models and optimization logic operate autonomously, adapting to changing conditions while reducing the operational complexity for users through automated decision-making.
Data Source
AI summary
A method includes predicting, for sets of input data, an input data number of a subsequent interval of a first interval using an input data number of the first interval and an input data number of a previous interval of the first interval set in a neural network inference optimization, determining the predicted input data number to be a batch size of the subsequent interval, determining whether pipelining is to be performed in a target device based on a resource state of the target device, and applying, to the target device, an inference policy including the determined batch size and a result of the determining of whether the pipelining is to be performed.