Neural Network Matrix-Vector Multiply Non-Zero Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The execution of matrix-vector multiply operations in neural networks on electronic devices is inefficient due to large memory transactions, high power consumption, and increased memory bandwidth usage, particularly in portable devices with constrained resources.
Innovation Solution
Identifying non-zero data positions in input vectors and fetching only the corresponding matrix values for these positions, thereby reducing unnecessary memory transactions and operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the weight matrix is fetched from main memory for matrix-vector multiply operations, then the operation can be performed with complete data, but memory bandwidth usage and power consumption increase significantly
Solution Approach 1:
The patent extracts and identifies only the non-zero elements from the input vector, then fetches only the corresponding weight values from main memory. This selective extraction approach maintains operational completeness while reducing memory bandwidth usage and power consumption by avoiding fetches for zero-valued positions.
Solution Approach 2:
Instead of fetching the entire weight matrix from main memory, the patent performs partial action by fetching only the necessary subset of weight values corresponding to non-zero input elements. This reduces memory transactions while ensuring sufficient data is retrieved for accurate computation.
2Measurement precision
If the entire weight matrix is fetched from main memory, then accurate matrix-vector multiply operations can be performed, but memory bandwidth usage increases
Solution Approach 1:
The system extracts the positions of non-zero elements from the input vector and uses these positions to selectively fetch only the corresponding weight values from main memory. This ensures accurate computations are performed on necessary data while minimizing memory bandwidth consumption by excluding unnecessary weight fetches.
3Productivity
If all matrix values are fetched from memory, then complete processing can be performed, but processing burden and memory requirements increase
Solution Approach 1:
The patent identifies and extracts only the non-zero positions from input vectors, then uses these extracted positions to selectively retrieve corresponding weight values from memory. This reduces the total number of memory transactions and processing operations required, thereby decreasing processing burden and memory requirements while maintaining complete and accurate neural network processing.
Data Source
AI summary
When performing a matrix-vector multiply operation for neural network processing, a set of one or more input vectors to be multiplied by a matrix of data values is scanned to identify data positions of the input vector(s) for which the data value is non-zero in at least one of the input vectors. For each of the data positions identified as having a non-zero value in at least one of the input vectors, the set of data values from the matrix of data values for that data position is fetched from memory and the matrix-vector multiply operation is performed using the data values for the input vectors for the data positions identified as being non-zero and the fetched set(s) of data values from the matrix of data values for those data position(s).