Deep Learning Computation Graph Fusion for Lower Cache Pressure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning computation graphs for complex neural networks are computationally expensive and time-consuming, leading to significant cache pressure due to inefficient operator fusion methods like fixed pattern and polyhedral-based loop fusion.
Innovation Solution
The method involves fusing memory-intensive operators into compute-intensive operators to create a new computation graph, which is then divided into sub-computation graphs, followed by further operator fusion within each sub-graph to optimize the computation process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional operator fusion methods (fixed pattern or polyhedral-based loop fusion) are used to optimize deep learning computation graphs, then the computation graph can be optimized, but the optimization process becomes computationally expensive and time-consuming, resulting in large cache pressure
Solution Approach 1:
The computation graph is divided into multiple sub-computation graphs based on operator types and data flow dependencies. This segmentation allows the optimization process to handle smaller, more manageable portions of the graph independently, reducing the overall computational complexity and time required for optimization while maintaining effectiveness.
Solution Approach 2:
Memory-intensive operators are extracted and fused into compute-intensive operators separately from the main optimization process. This extraction allows the optimization to focus on compute-intensive operations first, reducing the immediate computational burden and cache pressure during the optimization process.
2Device complexity
If memory-intensive operators are fused into compute-intensive operators, then the computation graph is simplified, but the optimization process still requires dividing the graph into sub-computation graphs to manage complexity
Solution Approach 1:
The computation graph is segmented into sub-computation graphs based on operator characteristics and data flow patterns. This segmentation enables parallel processing of optimization tasks across different sub-graphs, improving overall optimization throughput while keeping individual sub-graphs manageable in complexity.
Solution Approach 2:
Different fusion strategies are applied to different parts of the computation graph based on their specific characteristics. Compute-intensive operators receive one type of optimization while memory-intensive operators receive another, allowing each region to be optimized according to its local requirements without compromising overall efficiency.
3Quantity of substance
If the computation graph is divided into sub-computation graphs, then cache pressure is reduced, but additional processing steps are required for division and subsequent fusion operations
Solution Approach 1:
The computation graph is divided into sub-computation graphs that can be processed independently, reducing the amount of data that needs to be held in cache simultaneously. This segmentation directly reduces cache pressure while the modular nature of sub-graphs makes the additional processing steps more manageable and systematic.
Solution Approach 2:
The computation graph is pre-divided into sub-computation graphs before the main optimization process begins. This preliminary organization reduces cache pressure during subsequent optimization operations by ensuring that only relevant portions of the graph are actively processed in memory at any given time, rather than loading the entire graph.
Data Source
AI summary
Provided herein are apparatus and method for optimizing deep learning computation graph. The method includes obtaining a deep learning computation graph including compute-intensive operators and memory-intensive operators; fusing the memory-intensive operators into the compute-intensive operators to generate a new computation graph; dividing the new computation graph into sub-computation graphs; and fusing compute-intensive operators, in each of the sub-computation graphs, to generate an optimized computation graph. Other embodiments may also be disclosed and claimed.