Deep Learning Compiler Kernel Fusion for Neural Network Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training and inferencing using neural networks require significant memory and computing resources, which can be inefficiently utilized in existing systems.
Innovation Solution
A deep learning compiler that combines operations such as reduction, elementwise, and copy operations into a single software kernel, optimizing resource usage by generating a schedule that groups dependent reduction operations and rewrites higher-level operations into more efficient forms, such as reshape and sum reduction operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If operations are performed separately in existing systems, then implementation simplicity is maintained, but memory and computing resource usage increases
Solution Approach 1:
The patent combines multiple independent operations (reduction operations, elementwise operations, and copy operations) into a single unified software kernel. This merging eliminates redundant memory accesses and computing operations by performing all operations in one executed kernel, directly reducing computing resource usage and memory bandwidth requirements.
Solution Approach 2:
The unified software kernel serves multiple functions simultaneously - it performs reduction operations, elementwise operations, and copy operations within a single execution context. This multi-functionality allows the same kernel to handle diverse computational tasks without requiring separate specialized kernels for each operation type.
2Quantity of substance
If multiple separate operations are executed, then operational flexibility is maintained, but memory bandwidth and computing resources increase
Solution Approach 1:
By merging reduction, elementwise, and copy operations into a single kernel, the patent reduces the total quantity of memory operations. Intermediate results are kept in registers or shared memory within the kernel rather than being written to and read from global memory multiple times, thereby reducing overall memory usage and bandwidth requirements.
3Productivity
If operations are performed independently, then ease of implementation is maintained, but training and inferencing efficiency decreases
Solution Approach 1:
The deep learning compiler performs preliminary analysis and optimization of operation schedules before kernel generation. It identifies opportunities to combine operations and reorders them for optimal performance, preparing the operation graph in advance so that the resulting unified kernel executes efficiently during training and inferencing without requiring complex runtime decisions.
4Loss of energy
If higher-level operations are used directly, then programming simplicity is maintained, but resource efficiency decreases
Solution Approach 1:
The compiler transforms higher-level operations into more efficient lower-level operations by changing their computational parameters and representation. For example, it rewrites matrix operations into optimized tensor operations with specific stride and tiling parameters, and converts high-level reduction operations into efficient GPU reduction primitives, thereby improving resource efficiency while maintaining the same mathematical functionality.
Data Source
AI summary
Apparatuses, systems, and techniques to combine operations. In at least one embodiment, a processor causes two or more dependent reduction operations to be combined into a software kernel.