Neural Network Optimized Matrix Multiplication via Uniform Weight Patterns
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) are costly to implement due to their large model capacity, which is a challenge for mobile and on-device applications with limited storage, computation power, and energy consumption, and existing weight pruning methods often worsen inference computation performance due to unstructured sparsity in pruned weight matrices.
Innovation Solution
A method for neural network optimized matrix-matrix multiplication (NNMM) is introduced, where a neural network is retrained to produce a uniform pattern in weight coefficients, allowing for fewer multiplication operations by sharing results within blocks, and the matrix multiplication is optimized using a uniform pattern based GEMM operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks with large model capacity are used, then prediction performance is improved, but computational cost and energy consumption increase
Solution Approach 1:
The patent changes the structural parameters of the weight matrix by enforcing uniform patterns where multiple weight coefficients share the same value. This parameter transformation reduces the effective number of unique weights from O(N²) to O(N), directly lowering computational cost and energy consumption while preserving prediction performance through the reduced matrix multiplication operations.
Solution Approach 2:
The patent applies different uniform pattern configurations to different blocks or regions of the weight matrix. Each local region can have its own uniform pattern structure (e.g., row-uniform, column-uniform, or block-uniform), allowing optimized computation in each local area while maintaining overall network performance through the collective effect of all regions.
2Quantity of substance
If weight pruning is applied to reduce model size, then storage cost is reduced, but inference computation performance deteriorates due to unstructured sparsity
Solution Approach 1:
The patent segments the weight matrix into multiple blocks or regions, each enforced with uniform pattern constraints. This segmentation transforms the unstructured sparsity problem into structured uniform patterns, where each segment can be processed efficiently. The block-wise uniform pattern enforcement reduces model size while maintaining computation performance through regular, predictable patterns that enable optimized GEMM operations.
Solution Approach 2:
The patent fundamentally changes the weight distribution parameter from arbitrary real values to discrete uniform pattern values. This parameter change creates a structured representation where weights are organized into uniform blocks, reducing storage requirements and enabling faster computation through pattern-based optimization rather than handling unstructured sparse data.
3Productivity
If uniform pattern training is applied, then the number of multiplication operations is reduced, but training complexity increases
Solution Approach 1:
The patent applies uniform pattern constraints during the training phase as a preliminary action. By enforcing uniform patterns on weight matrices before and during training, the system prepares the model in advance to benefit from reduced multiplication operations during inference. This preliminary structuring of weights simplifies the inference computation while the training complexity overhead is managed through efficient gradient computation methods that respect the uniform pattern constraints.
Data Source
AI summary
According to embodiments, a method of neural network optimized matrix-matrix multiplication (NNMM) is performed by at least one processor and includes determining a first matrix of input coefficients, and determining a second matrix of weight coefficients of a neural network that is trained so that the second matrix has a predetermined uniform pattern, the predetermined uniform pattern having at least two of the weight coefficients that have a same value. The method further includes performing multiplication of the determined first matrix and the determined second matrix to determine output coefficients.