Fused Matrix Operations in Systolic Array Processors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based machine learning systems face challenges in computational requirements, power consumption, and memory bandwidth utilization, particularly when handling large matrices in deep neural networks.
Innovation Solution
The implementation of matrix operations using 2-dimensional data structures referred to as 'tiles' in computer hardware, which allows for efficient matrix multiplication and accumulation operations through fused instructions, reducing the need for memory writes and optimizing power usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If deep learning neural networks process large matrices using conventional methods, then computational accuracy is maintained, but memory bandwidth utilization increases and power consumption rises
Solution Approach 1:
The patent combines multiple separate operations (matrix multiplication, accumulation, and activation functions) into a single fused operation executed in one instruction. This merging eliminates intermediate memory writes and reads, reducing memory bandwidth utilization and power consumption while maintaining computational accuracy for deep learning neural network operations
Solution Approach 2:
The patent segments the large matrix processing into tile-based operations that can be executed efficiently in registers. By dividing matrices into manageable tiles and processing them through fused operations, the system reduces the need for frequent memory accesses while maintaining the ability to handle large-scale deep learning computations
2Measurement precision
If deep learning models are implemented with high computational requirements, then processing accuracy is maintained, but power consumption increases
Solution Approach 1:
The fused multiply-accumulate-activation operation combines multiple computational steps into a single instruction that executes atomically in hardware. This eliminates intermediate results that would require memory storage, reducing power consumption while maintaining the precision needed for deep learning through direct computation of the complete operation chain
Solution Approach 2:
The patent implements continuous computation by fusing operations that allow data to flow directly from multiplication through accumulation to activation function application without interruption or memory intervention. This continuous action maintains computational precision while minimizing the energy-intensive memory access cycles
3Productivity
If matrix operations are performed using conventional sequential methods, then computational correctness is ensured, but computational efficiency decreases
Solution Approach 1:
The patent merges matrix multiplication, accumulation, and activation into a single fused instruction that is verified by hardware logic to maintain mathematical correctness. This fusion achieves high computational efficiency through parallel execution while ensuring reliability through atomic operation design that prevents intermediate state errors
Solution Approach 2:
The patent replaces conventional software-based matrix operation sequences with dedicated hardware circuitry designed specifically for fused multiply-accumulate-activation operations. This substitution achieves superior computational efficiency through hardware parallelism while maintaining correctness through deterministic hardware logic design
Data Source
Figure 1A~1B
Figure 2(A)~2(C)
Figure 3
AI summary
Disclosed embodiments relate to deep learning implementations using systolic arrays and fused operations. In one example, a processor includes fetch and decode circuitry to fetch and decode an instruction having fields to specify an opcode and locations of a destination and N source matrices, the opcode indicating the processor is to load the N source matrices from memory, perform N convolutions on the N source matrices to generate N feature maps, and store results of the N convolutions in registers to be passed to an activation layer, wherein the processor is to perform the N convolutions and the activation layer with at most one memory load of each of the N source matrices. The processor further includes scheduling circuitry to schedule execution of the instruction and execution circuitry to execute the instruction as per the opcode.