Neural Network Matrix Multiplication Without Reshape or Transpose
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently processing complex data sets using neural networks, particularly in devices with low power and low performance, where real-time processing of large amounts of input data is required for neural networks.
Innovation Solution
A neural network apparatus and method that includes a processor capable of dividing initial weights in either the column or row direction based on the order of reshape and transpose operations relative to matrix multiplication, generating intermediate feature maps through matrix multiplication, and combining them to create a final feature map, thereby optimizing memory access and reducing computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the initial weight is divided into division weights, then the matrix multiplication operation can be performed without reshape and transpose operations, but the device complexity increases due to additional weight division steps
Solution Approach 1:
The patent segments the initial weight matrix into multiple division weights by dividing it in either the row direction or column direction. This segmentation allows the matrix multiplication to be performed without requiring reshape and transpose operations, thereby improving processing efficiency while the segmentation itself adds computational steps that increase device complexity
Solution Approach 2:
The patent performs weight division as a preliminary action before the matrix multiplication operation. By pre-dividing the initial weight into division weights, the system avoids the need for reshape and transpose operations during the main computation, improving productivity. However, this preliminary division action increases the overall computational overhead
2Device complexity
If reshape and transpose operations are performed before or after matrix multiplication, then the computational operations are reduced, but memory access patterns become less optimal
Solution Approach 1:
Instead of performing reshape and transpose operations before or after matrix multiplication, the patent inverts the approach by dividing the weight matrix into division weights. This inversion eliminates the need for reshape and transpose operations while also optimizing memory access patterns, as the division weights can be directly applied to the input feature map without requiring additional data rearrangement operations
3Productivity
If the weight matrix is divided in column direction, then intermediate feature maps are generated and concatenated, but the processing time increases due to additional concatenation operations
Solution Approach 1:
The patent segments the weight matrix into division weights by dividing in the column direction. This segmentation enables the generation of intermediate feature maps through matrix multiplication, and the final feature map is obtained by concatenating these intermediate maps. While this improves computational performance by avoiding reshape and transpose operations, the concatenation step adds processing time
4Productivity
If the weight matrix is divided in row direction, then intermediate feature maps are generated and summed element-wise, but the processing time increases due to additional sum operations
Solution Approach 1:
The patent segments the weight matrix into division weights by dividing in the row direction. This segmentation enables the generation of intermediate feature maps, and the final feature map is obtained by element-wise summing these intermediates. This approach improves computational performance by eliminating reshape and transpose operations, though the element-wise sum operation increases processing time
Data Source
AI summary
A neural network apparatus for performing a matrix multiplication operation includes a memory having at least one program stored therein and a processor to perform one or more operations by executing the at least one program. The processor can determine whether to divide an initial weight in one of a column direction and a row direction according to whether a reshape operation and a transpose operation are performed before or after a matrix multiplication operation and generate division weights by dividing the initial weight by a head count in the determined direction. Also, the processor can generate intermediate feature maps by performing a matrix multiplication operation between the input feature map and the division weights and generate a final feature map based on the intermediate feature maps.