Weight Matrix Formatting for Balanced Neural Network Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network weight matrix formatting methods fail to evenly distribute computation loads across processing elements, leading to prolonged operation times.
Innovation Solution
A method that determines row lengths for each row of the weight matrix based on the number of non-zero elements, rearranges rows and columns, and distributes them evenly across processing elements to generate formatted data, ensuring balanced computation loads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If weight matrix is formatted using CSR, CSC, or CISR methods to reduce memory capacity, then data size is reduced, but computation load cannot be evenly distributed to processing elements
Solution Approach 1:
The patent segments the weight matrix into multiple blocks and distributes these blocks across multiple processing elements. By dividing the matrix into manageable segments and assigning them to different PEs, the system achieves both compact storage (reducing data size) and balanced computation load distribution across the processing elements.
Solution Approach 2:
The patent applies different formatting strategies to different blocks of the weight matrix based on their local characteristics. Each block can be formatted independently according to its specific properties, allowing optimization for both storage efficiency and computation distribution while maintaining the overall structure needed for parallel processing.
2Quantity of substance
If weight matrix is formatted using CSR, CSC, or CISR methods, then memory capacity requirement is reduced, but operation time becomes long
Solution Approach 1:
By segmenting the weight matrix into blocks that can be independently processed and distributed across multiple PEs, the system reduces memory capacity requirements through efficient formatting while simultaneously enabling parallel execution. This segmentation allows operation time to be reduced despite the formatted structure, as multiple blocks can be processed concurrently.
Solution Approach 2:
The patent introduces a block dimension to the traditional formatting approaches. Instead of only considering row or column compression, it adds a spatial dimension by dividing the matrix into blocks that can be distributed across the processor array, thereby optimizing both memory usage and execution time through multi-dimensional organization.
3Ease of manufacture
If rows of weight matrix are distributed to processing elements without considering row length, then distribution is simple, but computation loads become uneven
Solution Approach 1:
The patent changes the distribution parameter from simple row index assignment to row length-aware assignment. By considering the number of non-zero elements (row length) as a key parameter in the distribution process, the system achieves balanced computation loads across PEs while maintaining a relatively simple distribution mechanism that builds upon existing formatting approaches.
Data Source
AI summary
A method formats a weight matrix in a current layer included in a neural network. The method includes calculating a row length for each row of the weight matrix based on a number of elements each of which has non-zero value; storing rearrangement information including result of sorting rows in the order of row lengths; performing a row transformation or the row transformation and a column transformation on the weight matrix using the rearrangement information; distributing rows of a transformed weight matrix to a plurality of processing elements (PEs); and generating formatted data including one or more group data each including values and column information being processed in each of the PEs.