Compiler Operation Fusion for Memory Bandwidth Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Memory bandwidth bottlenecks in data processing operations, particularly for convolution kernels in image processing and machine learning, hinder performance due to high memory access requirements.
Innovation Solution
A compiler that automatically identifies sequences of operations for fusion, determines subdivisions of the final output, and generates code to fuse operations, reducing memory bandwidth by storing intermediate data in local memory instead of external memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If convolution kernels are processed using separate operations with external memory storage, then memory capacity is sufficient, but memory bandwidth becomes a bottleneck reducing processing performance
Solution Approach 1:
The patent merges multiple convolution operations into a single fused operation, allowing intermediate results to be computed and stored in local memory without writing to external memory. This combining of operations reduces the total volume of data transferred between external memory and processing units, directly addressing the memory bandwidth bottleneck while maintaining sufficient memory capacity for the computation.
Solution Approach 2:
The patent implements a nested memory hierarchy where local memory is embedded within the processing unit and contains intermediate results of fused operations. This nested structure allows the system to keep frequently accessed intermediate data in fast local memory, reducing reliance on external memory bandwidth while maintaining adequate storage capacity through the hierarchical arrangement of memory spaces.
2Quantity of substance
If operations are fused to reduce memory bandwidth, then memory access is reduced, but code complexity increases requiring automatic code generation
Solution Approach 1:
The patent employs an automatic code generation system that analyzes the computation graph, identifies fusion opportunities, and generates optimized code without manual intervention. This self-service approach to code generation handles the complexity of fused operations automatically, allowing the system to reduce memory bandwidth through operation fusion while the code generation tool manages the resulting complexity, eliminating the need for manual code optimization efforts.
3Quantity of substance
If intermediate results are stored in local memory through tiling, then memory bandwidth is reduced, but local memory capacity must be sufficient to hold subdivided data
Solution Approach 1:
The patent divides the computation and data into tiles or subdivisions that fit within local memory capacity. By segmenting the overall computation into smaller manageable chunks, the system can store intermediate results of each tile in local memory without exceeding its capacity limits. This segmentation approach reduces memory bandwidth usage for each tile while ensuring that local memory capacity requirements remain achievable through appropriate tile size selection.
Data Source
AI summary
A compiler and a method of compiling code that reduces memory bandwidth when processing code on a computer are provided herein. In one embodiment, the method includes: (1) automatically identifying a sequence of operations for fusing, wherein the sequence of operations correspond to instructions from a source code, (2) determining subdivisions of a final output of the sequence of operations, (3) determining input data and intermediate operations needed to obtain a final subdivision output for each of the subdivisions and (4) automatically generating code to fuse the sequence of operations employing the subdivisions, wherein the automatically identifying and the automatically generating are performed by a processor.