Skipping Inconsequential Matrix Operations in Neural Network Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current hardware accelerators for neural networks inefficiently handle matrix multiplication due to the presence of zeros or close-to-zero values in matrices, leading to wastage of power and compute cycles, as they perform operations in dense formats without considering sparsity.
Innovation Solution
The introduction of instructions such as TILENZFMA and TILENZMUL that detect and skip inconsequential multiplications by identifying near-zero values, allowing for dynamic detection or pre-tagging of elements to be skipped, thereby enhancing performance and power conservation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hardware accelerators perform matrix multiplication in dense format, then compute throughput is maximized, but power consumption increases due to processing zero-valued elements
Solution Approach 1:
The patent extracts and removes zero-valued or near-zero elements from matrix operations before processing. By identifying and eliminating inconsequential multiplicands that would produce zero or near-zero products, the system avoids allocating compute cycles and power to these useless operations, thereby resolving the contradiction between maintaining throughput and reducing power consumption.
Solution Approach 2:
The patent applies partial action by performing matrix multiplication only on non-zero elements rather than processing the entire dense matrix. The instructions selectively execute multiply-accumulate operations only where needed, skipping zero-valued elements entirely. This partial processing approach maintains necessary computational throughput while eliminating wasteful power consumption on inconsequential operations.
2Reliability
If hardware accelerators process all elements in dense matrix format, then computation completeness is ensured, but hardware resource utilization becomes inefficient
Solution Approach 1:
The patent applies preliminary action by pre-processing matrices to identify and tag zero-valued or near-zero elements before the main multiplication operation. Instructions such as TILENZFMA and TILENZMUL use pre-computed sparsity masks or threshold comparisons to mark inconsequential elements, allowing the hardware accelerator to skip these elements during execution. This preliminary identification ensures computation completeness is maintained for non-zero elements while improving hardware resource utilization.
Solution Approach 2:
The patent applies local quality by treating different elements of the matrix differently based on their values. Instead of uniformly processing all elements in dense format, the system applies selective processing where zero-valued or near-zero elements are skipped while non-zero elements receive full processing attention. This local differentiation optimizes hardware resource utilization without compromising the reliability of meaningful computations.
3Ease of operation
If matrix operations include zero-valued elements, then data representation is simplified, but compute cycles are wasted generating inconsequential products
Solution Approach 1:
The patent applies the skipping principle by enabling hardware accelerators to detect and skip over zero-valued or near-zero elements during matrix multiplication. Instructions like TILENZFMA and TILENZMUL incorporate mechanisms to identify inconsequential multiplicands and bypass the multiplication and accumulation steps for these elements. This skipping approach eliminates wasted compute cycles while maintaining simplified dense data representation for the remaining non-zero elements.
Data Source
AI summary
Disclosed embodiments relate to systems and methods to skip inconsequential matrix operations. In one example, a processor includes decode circuitry to decode an instruction having fields to specify an opcode and locations of first source, second source, and destination matrices, the opcode indicating that the processor is to multiply each element at row M and column K of the first source matrix with a corresponding element at row K and column N of the second source matrix, and accumulate a resulting product with previous contents of a corresponding element at row M and column N of the destination matrix, the processor to skip multiplications that, based on detected values of corresponding multiplicands, would generate inconsequential results; scheduling circuitry to schedule execution of the instruction; and execution circuitry to execute the instructions as per the opcode.