Neural Network Layer Grouping for Hardware Pass Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) are computationally complex to execute efficiently on neural network accelerators, requiring innovative methods to optimize their execution by dividing layers into manageable chunks that align with hardware constraints.
Innovation Solution
A method is described to divide a neural network into layer groups and further into chunks of operations executable in a hardware pass, using split parameters to minimize a neural network loss function, which represents performance metrics such as execution cycles or bandwidth, based on hardware constraints like buffer sizes and operation types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the neural network is executed directly on the neural network accelerator without division, then the execution structure is simple, but the hardware constraints (buffer sizes, operation types) cannot be satisfied and execution efficiency is poor
Solution Approach 1:
The patent divides the neural network into multiple layer groups, where each layer group contains a sequence of layers that can be executed in a single hardware pass. This segmentation allows the network to be processed in manageable chunks that fit within hardware constraints while maintaining overall execution efficiency.
Solution Approach 2:
The patent introduces dynamic split parameters that can be adjusted to optimize the division of layer groups. These parameters allow the execution structure to adapt to different hardware configurations and network architectures, balancing complexity and efficiency based on specific requirements.
2Productivity
If the neural network is divided into many small layer groups, then the hardware pass utilization is improved, but the number of hardware passes increases and execution time increases
Solution Approach 1:
The patent uses optimization techniques to find the optimal split parameters that divide the network into layer groups of appropriate size. By adjusting these parameters, the system achieves the right balance between hardware pass utilization and the number of passes required, minimizing total execution time.
Solution Approach 2:
The patent employs a loss function that evaluates the performance metric (such as execution time or bandwidth) based on the division into layer groups. This feedback mechanism guides the optimization process to find divisions that maximize hardware utilization while minimizing the number of hardware passes required.
3Productivity
If the split parameters are optimized for minimum execution cycles, then the performance metric is improved, but the complexity of determining split parameters increases
Solution Approach 1:
The patent performs preliminary analysis to establish the relationship between split parameters and performance metrics before actual execution. By pre-determining optimal parameter ranges and using loss functions to guide the optimization, the system reduces the complexity of real-time parameter determination while maintaining high performance.
Data Source
AI summary
Methods of dividing a neural network into chunks of operations executable in a hardware pass of hardware to execute a neural network. The layers of the neural network are divisible into layer groups that comprise a sequence of layers executable in the same hardware pass of the hardware. Each layer group is divisible into chunks of operations executable in a hardware pass of the hardware. The chunks for a layer group are defined by split parameters. A layer group loss function is obtained that represents a performance metric associated with executing a layer group on the hardware as a function of the split parameters and neural network architecture parameters for the layer group. A neural network loss function is generated based on the layer group loss function that represents the performance metric associated with executing the neural network on the hardware; and the split parameters for the one or more layer groups are selected that minimize the neural network loss function under constraints imposed by the hardware.