Distributed Parser Circuitry for GPU Workload Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed architectures for GPUs face challenges in efficiently distributing compute workloads across multiple processing elements, leading to suboptimal performance and power consumption due to reduced information availability and potential overburdening of communication fabrics.
Innovation Solution
A distributed workload parser architecture that includes a global workload parser and multiple distributed workload parsers, which communicate via a communications fabric to efficiently manage and distribute compute workloads, using batch iterators and workgroup iterators to coordinate the execution of compute kernels across multiple shader cores, reducing data transmission and optimizing load balancing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If a centralized workload parser is used to distribute compute workloads, then information availability is high, but communication fabric overburdening occurs and data transmission increases
Solution Approach 1:
The workload parser is segmented into a master workload parser and multiple distributed workload parsers. The master parser retains full kernel state information while distributed parsers receive only batch information and coordinate their own workgroup distribution, reducing communication fabric load while maintaining information availability where needed.
Solution Approach 2:
The system introduces a hierarchical dimension to the parser architecture, with master parsers operating at a global level and distributed parsers operating at local levels. This dimensional organization allows information to be available globally when needed while distributing the burden of workload management locally, reducing overall communication fabric overburdening.
2Productivity
If compute workloads are distributed across multiple processing elements, then compute power utilization improves, but load balancing becomes difficult and performance becomes suboptimal
Solution Approach 1:
Distributed workload parsers send completion status and workload information back to the master parser, which uses this feedback to dynamically adjust batch assignments and maintain optimal load balancing across processing elements, ensuring high compute power utilization while managing complexity.
Solution Approach 2:
The system implements dynamic batch size adjustments where the master parser can modify the number of workgroups in batches based on real-time processing element status, enabling flexible load balancing that adapts to changing workload conditions and maintains optimal performance.
3Device complexity
If batch sizes are fixed in distributed parser architecture, then system complexity is reduced, but compute efficiency decreases due to inability to optimize for different workload scenarios
Solution Approach 1:
The system implements dynamic batch size adjustments where the master parser can modify the number of workgroups in batches based on real-time processing element status and workload characteristics, enabling flexible optimization of compute efficiency without permanently increasing system complexity.
Solution Approach 2:
The batch size parameter is made adjustable and dynamic rather than fixed, allowing the system to optimize compute efficiency for different workload scenarios by changing the batch size parameter as needed while maintaining the same underlying parser architecture.
Data Source
AI summary
Techniques are disclosed relating to distributing work from compute kernels using distributed parser circuitry. In some embodiments, a master parser is configured to communicate with distributed parsers over a communications fabric. In some embodiments, the master parser generates batches of compute workgroups from a compute kernel and assigns batches to ones of the distributed workload parser circuits. In some embodiments, based on an indicated number of sequential workgroups to send to the same distributed workload parser, the master parser selects a distributed parser to receive a set of batches and avoids selecting a distributed workload parser whose queue would be filled by the batches. In some embodiments, this may avoid stalling for a kernel terminate command. In some embodiments, the master parser may adjust a batch size to avoid filling a distributed parser's queue. In some embodiments, the distributed parsers may use similar techniques when sending work to shader units.