Neural Network Kernel Fusion to Reduce Memory Access Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing compute kernels for neural networks are inefficiently optimized through manual programming and automatically generated kernels often result in multiple kernels per cell, leading to repeated memory access and hindered performance due to increased overhead.
Innovation Solution
Automated optimization of neural network graphs to reduce the number of kernels by adjusting operations, such as replacing matrix-vector and matrix-matrix multiplications with sequences of reshape, element-wise multiplication, and sum reduction operations, and moving slice operations, resulting in a single optimized kernel for each cell.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual programming is used to optimize compute kernels, then optimization quality may be improved, but time consumption and efficiency deteriorate
Solution Approach 1:
The system performs automatic kernel generation and optimization without manual intervention. The compiler automatically analyzes the computational graph, identifies optimization opportunities, and generates optimized kernels, making the system self-sufficient in the optimization process.
Solution Approach 2:
The patent replaces manual programming mechanics with an automated compilation system. Instead of manually writing and optimizing kernel code, the system uses an automated compiler that analyzes computational graphs and generates optimized kernels through algorithmic processes.
2Extent of automation
If multiple separate kernels are generated for a single neural network cell, then automatic kernel generation is achieved, but performance deteriorates due to repeated memory access
Solution Approach 1:
The patent merges multiple separate kernels into a single unified kernel for each neural network cell. By combining the functions of multiple kernels that previously performed separate operations on input tensors, the system eliminates repeated memory access and improves performance while maintaining automatic generation.
Solution Approach 2:
The generated kernel is designed to perform multiple functions within a single execution unit. It can handle different operations (such as convolution, activation, and normalization) in sequence without requiring separate kernel launches, making the kernel multi-functional and reducing overhead.
3Adaptability or versatility
If multiple kernels are used for neural network operations, then operation coverage is improved, but device complexity increases due to multiple kernel management
Solution Approach 1:
The patent creates a universal kernel that can perform multiple neural network operations through a single execution unit. This multi-functional kernel reduces the number of separate kernels needed while maintaining comprehensive operation coverage, thereby simplifying kernel management.
Data Source
AI summary
Apparatuses, systems, and techniques to automatically generate a reduced number of compute kernels for performing operations of one or more neural networks. In at least one embodiment, one or more operations of one or more neural network graph nodes of the one or more neural network are automatically adjusted to generate an optimized one or more operations that are compiled to generate the reduced number of compute kernels.