Matrix Multiplication Tile Packing for Concurrent Neural Network Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face reduced throughput and wasted compute cycles due to matmul operations that do not fully utilize processing engine arrays, leading to idle sections when smaller tensors are processed.
Innovation Solution
A compiler with matmul packing capability that automatically identifies instruction-level parallelism, reorders matmul instructions for concurrent execution, and assigns them to tiles on the processing engine array, resolving conflicts to ensure schedulable and allocatable vectorized IR.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If matmul operations are executed on processing engine arrays, then computational throughput is improved, but smaller tensors cause idle sections and wasted compute cycles
Solution Approach 1:
The patent combines multiple smaller matmul operations into a single packed matmul operation by identifying instruction-level parallelism and reordering instructions to execute multiple operations concurrently on the processing engine array, thereby eliminating idle sections and maximizing throughput
Solution Approach 2:
The patent segments the processing engine array into multiple tiles and assigns different matmul operations to different tiles, allowing simultaneous execution of multiple operations across the array while fully utilizing all processing engines
2Productivity
If matmul instructions are reordered for concurrent execution, then resource utilization is improved, but instruction reordering complexity increases
Solution Approach 1:
The patent performs preliminary analysis during code generation to identify instruction-level parallelism and determine optimal tile assignments before execution, storing this information in metadata so that runtime reordering is simplified and efficient
3Productivity
If multiple matmul operations are packed together, then throughput is improved, but conflicts in tile assignment may arise
Solution Approach 1:
The patent assigns different tile positions to different packed matmul operations based on their specific requirements and dependencies, ensuring that each operation receives appropriate resource allocation while maintaining overall schedulability and avoiding conflicts
Data Source
AI summary
A technique for arranging matrix multiplications for concurrent execution in an integrated circuit device may include obtaining a representation of a data dependency graph of a neural network model. The data dependency graph may include having an accumulation group (AG) pack of accumulation groups (AGs), in which each of the AGs has one or more matrix multipartition instructions. A representation of a memory location base partition constraint graph of the AG pack can be generated, and an AG row group constraint graph can be generated based on the memory location base partition constraint graph. The AGs of the AG pack can then be assigned to tiles in an integrated circuit device based on the AG row group constraint graph.