Dynamic Thread Allocation for DNN Inference Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing DNN serving systems face challenges in efficiently handling high request rates with low response latency, as the benefits of intra-operator parallelism diminish with increasing thread counts.
Innovation Solution
A thread optimization system that estimates batch sizes and automatically determines the optimal configuration of inference instances, threads per instance, and sub-batch sizes using a dynamic programming algorithm to minimize average per-batch latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If intra-operator parallelism is used to process batches with increasing thread counts, then inference latency is reduced, but the improvements in latency diminish as thread count increases
Solution Approach 1:
The patent segments the batch processing into multiple inference instances, each handling a portion of the batch. Instead of using one large instance with many threads, the system creates multiple smaller instances that can be processed in parallel, thereby avoiding the diminishing returns of intra-operator parallelism while maintaining low latency and improving throughput.
Solution Approach 2:
The patent transitions from a single-dimension approach (increasing threads within one instance) to a multi-dimensional approach by introducing the number of inference instances as an additional dimension. This allows the system to optimize both latency and throughput by adjusting not just thread count but also instance count and batch size per instance.
2Productivity
If multiple inference instances are used to maximize throughput, then system throughput is improved, but per-batch latency increases
Solution Approach 1:
The patent implements dynamic configuration adjustment, allowing the system to adaptively change the number of inference instances, threads per instance, and batch size per instance based on workload characteristics. This dynamic approach enables the system to optimize for throughput when appropriate while maintaining low latency when needed, rather than being fixed in one configuration.
Solution Approach 2:
The patent systematically varies key parameters including the number of inference instances, threads per instance, and batch size per instance to identify optimal configurations. By changing these parameters based on workload demands, the system can achieve both high throughput and low latency as needed.
3Adaptability or versatility
If users manually configure thread allocation for model instances, then flexibility is provided, but optimal latency and throughput cannot be achieved without extensive tuning
Solution Approach 1:
The patent implements self-service through automated configuration optimization. The system automatically determines optimal settings for the number of inference instances, threads per instance, and batch size per instance based on workload characteristics, eliminating the need for users to manually tune these parameters while still providing the flexibility to adjust configurations when needed.
Solution Approach 2:
The patent incorporates feedback mechanisms that monitor system performance and workload characteristics, using this information to automatically adjust configuration parameters. This feedback loop enables the system to continuously optimize for both latency and throughput without requiring manual intervention or extensive user tuning.
Data Source
AI summary
Systems and methods for optimizing thread allocation in a model serving system include estimating a batch size for inference requests. An optimal configuration is then determined that defines a number of inference instances, a number of threads per inference instance, and a sub-batch size per inference instance for processing a batch of inference requests of the batch size using intra-operator parallelism that minimizes average per-batch latency. The optimal configuration is determined with reference to a plurality of predetermined model profiles that define single-inference average batch latencies for different combinations of thread counts and batch sizes, the predetermined model profiles being used as input to a dynamic programming algorithm that identifies optimal configurations that minimize the average per-batch latency.