Dynamic Accelerator Partitioning for AI Batch Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for processing AI models face inefficiencies in resource utilization and throughput due to inadequate partitioning and scheduling of accelerators, leading to increased latency and idle time, especially when handling batches of varying sizes.
Innovation Solution
An electronic device with processors and memory that partitions an accelerator into multiple sizes based on resource utilization, determines correspondences between batch and partition sizes, and schedules batches to partitions based on predicted execution times to optimize resource utilization and meet latency constraints, using a neural network model for processing time determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If the accelerator is partitioned into fixed sizes, then the device complexity is reduced, but the resource utilization deteriorates when handling batches of varying sizes
Solution Approach 1:
The patent implements dynamic partitioning where the accelerator is divided into multiple partitions of different sizes based on real-time workload characteristics. The system adjusts the number and size of partitions dynamically according to batch size requirements, transitioning from static fixed partitions to adaptive dynamic partitions that optimize resource utilization for varying workload conditions.
Solution Approach 2:
The accelerator is segmented into multiple independent partitions of different sizes, allowing flexible allocation of processing resources. Each partition can independently handle batches of appropriate size, enabling fine-grained resource management and preventing both over-provisioning and under-provisioning of processing capacity.
2Productivity
If larger partitions are used to process larger batches, then the processing capacity is improved, but the latency increases for smaller batches
Solution Approach 1:
The patent applies local quality by assigning different partition sizes to different workload requirements. Small batches are routed to small partitions optimized for low latency, while large batches are routed to large partitions optimized for high throughput. This ensures that each batch size receives processing resources matched to its specific requirements, minimizing latency for small batches while maintaining high capacity for large batches.
3Loss of energy
If the accelerator is dynamically partitioned based on batch sizes, then the resource utilization is improved, but the device complexity increases
Solution Approach 1:
The system performs preliminary analysis of workload characteristics and batch size distributions before executing processing tasks. By predicting future workload patterns and pre-configuring appropriate partition configurations, the system avoids complex real-time decision-making during execution, reducing operational complexity while maintaining high resource utilization through proactive adaptation.
4Ease of operation
If batches are scheduled without considering execution time requirements, then the scheduling simplicity is maintained, but the latency constraints are violated
Solution Approach 1:
The patent implements feedback mechanisms that monitor actual execution times and latency performance, using this information to adjust scheduling decisions. The system continuously learns from performance data and adapts partition assignments and scheduling policies to meet latency constraints while maintaining operational simplicity through automated closed-loop control rather than complex manual scheduling rules.
Data Source
AI summary
An electronic device and method for partitioning an accelerator and scheduling batches are disclosed. An electronic device includes one or more processors, and a memory storing instructions configured to cause the one or more processors to, for a first partitioning of an accelerator into partitions of different sizes, based on resource utilization of the partitions batch of different sizes input to the partition, determine correspondences between the sizes of the batches and the sizes of the partitions in the first partitioning, determine numbers of partitions for the respective determined sizes of the partitions based on the correspondences between the sizes of the batches and the sizes of the partitions in the first partitioning, and partition the accelerator into a second partitioning based on the determined numbers of the respective sizes of the partitions.