Fixed Point Matrix Multiply Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural network models using fixed point computations on mobile devices require excessive memory, leading to reduced performance due to the need for higher precision intermediate results during matrix multiplication operations.
Innovation Solution
Implementing partial matrix multiplication using blocks of fixed point data, downconverting intermediate results to lower precision, and using saturating addition to cap output precision, thereby reducing memory requirements without significant accuracy loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full precision fixed point data is used for intermediate results in matrix multiplication, then computational accuracy is maintained, but memory requirements increase significantly
Solution Approach 1:
The patent segments the matrix multiplication process into multiple partial multiplications, where each partial multiplication operates on blocks of data with lower precision intermediate results. By dividing the computation into segments that can be accumulated incrementally, the system maintains final accuracy while reducing peak memory requirements for intermediate values.
Solution Approach 2:
The patent dynamically changes the precision parameter of intermediate results during computation. Intermediate values are stored at lower precision (reducing memory usage) while the final accumulation and output restore full precision. This parameter transformation allows the system to trade temporary precision for memory efficiency without compromising final computational accuracy.
2Measurement precision
If extra memory is allocated for fixed point computations, then computational accuracy is improved, but performance speed decreases
Solution Approach 1:
By segmenting the matrix multiplication into multiple smaller partial multiplications that can be performed with lower precision intermediates, the patent reduces memory access requirements. This segmentation enables faster computation by minimizing slow memory accesses while maintaining accuracy through incremental accumulation of results.
Solution Approach 2:
The patent performs partial matrix multiplications with lower than full precision intermediate results. By accepting temporarily reduced precision in intermediate steps (partial action) and restoring full precision only in the final output, the system achieves faster performance without sacrificing ultimate computational accuracy.
3Quantity of substance
If lower precision is used for intermediate results, then memory requirements are reduced, but computational accuracy may be compromised
Solution Approach 1:
The patent performs preliminary partial multiplications with lower precision intermediate results, accumulating these partial products incrementally. By preparing and accumulating multiple partial results at reduced precision before final combination, the system reduces memory requirements during intermediate storage while ensuring final accuracy is restored through proper accumulation and rounding.
Solution Approach 2:
The patent incorporates feedback mechanisms through saturating addition and proper rounding operations that monitor and correct precision loss during intermediate calculations. The accumulation process continuously feeds back into the computation, ensuring that errors from lower precision intermediates are minimized and the final result maintains the required computational accuracy.
Data Source
Figure 1~2
Figure 3A
Figure 3B
AI summary
Embodiments include computing devices, apparatus, and methods implemented by the apparatus for memory reduction for fixed point matrix multiply on a computing device. The computing device may implement a partial matrix multiplication using a first block of fixed point data of a first matrix and a second block of fixed point data of a second matrix using full precision resulting in a first intermediate result. The computing device may down convert the first intermediate result by converting fixed point data of the first intermediate result to fixed point data using lower precision resulting in a first down converted intermediate result.