Masked GEMM Instruction for Neural Network Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computer processors are inefficient in performing masked matrix multiplication operations required for neural network pruning, leading to suboptimal performance and power consumption during weight update steps in deep learning algorithms.
Innovation Solution
A new matrix-multiply instruction, GEMM (A, B, W, omask), is introduced, where A and B are input matrices, W is the output matrix, and omask is an output mask specifying which elements to zero in the resulting matrix, allowing for power-gating of multipliers and adders based on the output mask to enhance efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If traditional matrix multiplication is performed without masking, then computational completeness is maintained, but power consumption and computational overhead increase for pruned neural networks
Solution Approach 1:
The patent extracts and removes the computation of elements that will be zeroed out by the output mask. By identifying which output elements will be zeroed in advance, the system eliminates the unnecessary multiplication and addition operations for those elements, thereby reducing power consumption and computational overhead while maintaining the correctness of the final result.
Solution Approach 2:
The patent applies partial action by performing only the necessary computations for non-zeroed output elements. Instead of computing all elements and then zeroing some, the system computes only the subset of elements that need to be calculated, reducing overall computational effort while achieving the same effective result.
2Loss of energy
If masked matrix multiplication is implemented, then power consumption is reduced, but instruction set complexity increases
Solution Approach 1:
The patent implements a universal masked GEMM instruction that can handle both pruned and non-pruned neural network operations. The same instruction format and execution logic work whether or not a mask is provided, allowing the hardware to be used for multiple purposes without requiring separate dedicated instructions for masked operations.
Solution Approach 2:
The patent introduces an output mask parameter as an intermediary that controls the computation process. This mask serves as a mediator between the input matrices and the output matrix, indicating which elements should be zeroed. By using this intermediary, the system can selectively disable computations without requiring complex control logic in the instruction set.
3Productivity
If all matrix elements are computed and then zeroed, then implementation simplicity is maintained, but computational overhead and execution time increase
Solution Approach 1:
The patent applies preliminary action by determining which output elements will be zeroed before performing the matrix multiplication. The output mask is prepared in advance, allowing the execution unit to skip computations for elements that will be zeroed, rather than computing all elements and then zeroing them afterward. This preliminary identification of zeroed elements reduces unnecessary computational work.
Data Source
AI summary
An apparatus and method for a masked multiply instruction to support neural network pruning operations. For example, one embodiment of a processor comprises: a decoder to decode a matrix multiplication with masking (GEMM) instruction identifying a destination matrix register to store a result, and source registers storing an A-matrix, a B-matrix, and a matrix mask; execution circuitry to execute the GEMM instruction, the execution circuitry to multiply a plurality of B-matrix elements with a plurality of A-matrix elements, each of the B-matrix elements associated with a mask value in the matrix mask, wherein if the mask value is set to a first value, then the execution circuitry is to multiply the B-matrix element with one or more of the A-matrix elements to generate a first partial result, and if the mask value is set to a second value, then the execution circuitry is to multiply an alternate B-matrix element with a one or more of the A-matrix elements to generate a second partial result.