Hierarchical GEMM Control for Dynamic Resource Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GEMM operation tiling and execution methods on accelerators are inflexible and inefficient, leading to suboptimal resource utilization and bandwidth degradation due to fixed control schemes that do not account for varying GEMM sizes and resource capacities, resulting in wasted resources and reduced performance.
Innovation Solution
A programmable and hierarchical control system for GEMM operation tiling and execution, utilizing a global controller and local controllers coupled via queues to dynamically determine execution manners and allocate computation units, allowing for flexible scheduling and efficient data movement based on available resources and memory subsystems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If fixed control schemes are used for GEMM operation tiling and execution, then device complexity is reduced, but adaptability to varying GEMM sizes and resource capacities deteriorates
Solution Approach 1:
The patent implements a hierarchical control system where a global controller dynamically determines execution manners based on GEMM operation sizes and resource capacities, while local controllers adaptively manage data movement. This dynamic structure allows the system to adjust to varying workloads and resource availability, resolving the contradiction between maintaining simple fixed control schemes and achieving adaptability to different GEMM sizes and resource capacities.
Solution Approach 2:
The control system is segmented into a global controller and multiple local controllers. The global controller handles high-level execution manner determination and computation unit allocation, while local controllers handle specific data movement tasks. This segmentation allows each controller to operate with appropriate complexity for its function, achieving overall adaptability without requiring the entire system to be highly complex.
2Ease of operation
If fixed tiling methods are used, then ease of operation is improved, but resource utilization deteriorates
Solution Approach 1:
The system dynamically determines execution manners and partitions GEMM operations based on available resources and operation sizes. This dynamic approach automatically optimizes resource utilization without requiring manual intervention, maintaining ease of operation while significantly improving productivity through adaptive resource allocation and parallel execution strategies.
3Device complexity
If simple control structures are used, then device complexity is reduced, but execution efficiency deteriorates
Solution Approach 1:
The control structure is segmented into a global controller that makes high-level decisions about execution manners and computation unit allocation, and local controllers that handle specific data movement operations. This segmentation distributes complexity across multiple components, each with specialized functions, improving execution efficiency through coordinated parallel operations while keeping individual controller structures relatively simple.
Solution Approach 2:
Queues are introduced as intermediary structures between the global controller and local controllers. These queues buffer and manage the flow of control signals and data movement requests, enabling efficient coordination without requiring complex direct communication pathways, thus maintaining simpler controller structures while achieving high execution efficiency.
4Productivity
If parallel execution is implemented, then productivity is improved, but device complexity increases
Solution Approach 1:
The parallel execution system is segmented into independent local controllers, each managing specific data movement tasks for different computation units. This segmentation allows multiple data movement operations to proceed in parallel without requiring a single complex controller to manage all operations, thus improving productivity while keeping individual controller components relatively simple.
Data Source
AI summary
The present disclosure relates to a method for controlling execution of a GEMM operation on an accelerator comprising multiple computation units, a first memory device, and a second memory device. The method comprises determining an execution manner of the GEMM operation, the execution manner comprising partition information of the GEMM operation and computation unit allocation information of the partitioned GEMM operation; generating one or more instructions to compute the partitioned GEMM operation on one or more allocated computation units; and issuing the one or more instructions to at least one of a first queue and a second queue, which enables at least one of a first local controller and a second local controller to execute the one or more instructions, wherein the first local controller and the second local controller are configured to control data movement between the computation units, the first memory device, and the second memory device.