Matrix Multiplication Optimization Using Vector Registers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network models face inefficiencies in resource utilization due to irregular sparsity in weight matrices, leading to unnecessary computing power usage and reduced accuracy from pruning strategies, particularly in matrix multiplication operations.
Innovation Solution
The method involves partitioning the weight matrix into row-dominant and column-dominant sections and loading non-zero elements into vector registers with variable lengths, optimizing matrix multiplication by processing only non-zero elements and utilizing their distribution patterns to minimize computational operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If pruning strategies are applied to reduce model size, then resource utilization improves, but model accuracy deteriorates
Solution Approach 1:
The weight matrix is segmented into multiple sections based on the distribution pattern of non-zero elements. Each section is processed independently with optimized loading strategies, allowing the system to maintain accuracy by preserving important non-zero elements while reducing overall model size through selective processing.
Solution Approach 2:
Different sections of the weight matrix are treated differently based on their local characteristics. Sections with higher density of non-zero elements are processed with different strategies than sparse sections, optimizing the balance between model size reduction and accuracy preservation for each local region.
2Ease of operation
If all elements are loaded into vector registers for matrix multiplication, then computational operations are simplified, but computing power is wasted on zero elements
Solution Approach 1:
The method extracts only the non-zero elements from the weight matrix and loads them into vector registers, eliminating the unnecessary computation on zero elements. This extraction is performed by identifying the distribution pattern of non-zero elements and selectively loading only those that contribute to the computational result.
Solution Approach 2:
Instead of loading all elements (excessive action), the system loads only the necessary non-zero elements (partial action). This partial loading approach reduces computing power consumption while maintaining the ability to perform the required matrix multiplication operations.
3Productivity
If non-zero elements are loaded into vector registers, then computing efficiency improves, but memory access complexity increases
Solution Approach 1:
The system performs preliminary analysis of the weight matrix to identify the distribution pattern of non-zero elements before the actual matrix multiplication. This preliminary action creates a mapping of memory addresses to vector register locations, which simplifies the subsequent memory access during computation by pre-organizing the data layout.
4Quantity of substance
If variable length vector registers are used, then resource utilization improves, but hardware complexity increases
Solution Approach 1:
The system dynamically adjusts the number of vector registers and their lengths based on the specific characteristics of each matrix section. Rather than using fixed-length registers for all operations, the variable length configuration allows optimal resource utilization for each section while managing hardware complexity through software-controlled register allocation.
Data Source
AI summary
Methods and devices, the method including receiving a matrix of a neural network model; classifying at least a portion of the matrix as a first section based on a first distribution pattern of non-zero elements of the portion of the matrix; and identifying memory addresses of the non-zero elements in the first section of the matrix for loading, according to a first order determined based on the first distribution pattern, the non-zero elements in the first section into one or more vector registers.