Tensor Processor Scaling for Faster Matrix Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware accelerators inefficiency in performing matrix operations due to separate application of scale factors at SIMD or SIMT engines, leading to bottlenecks in matrix multiplication processes.
Innovation Solution
Performing both matrix multiplication and parameter scaling at the tensor unit within a hardware accelerator, reducing the workload of tile vector processors by applying scale factors directly at tensor processors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If scale factors are applied separately at SIMD or SIMT engines, then matrix multiplication can be performed, but processing efficiency deteriorates due to bottlenecks
Solution Approach 1:
The patent merges the scale factor application operation with the matrix multiplication operation by implementing the scaling function within the tensor processor itself. This integration eliminates the need for separate SIMD/SIMT engines to handle scaling, thereby removing the processing bottleneck and improving overall efficiency.
Solution Approach 2:
The tensor processor is designed to perform multiple functions: it both executes matrix multiplication and applies scale factors to the results. This multi-functionality consolidates what previously required separate specialized units, reducing system complexity and improving throughput.
2Adaptability or versatility
If separate SIMD or SIMT engines are used for scaling, then scaling can be performed, but device complexity increases
Solution Approach 1:
The patent combines the scaling functionality into the existing tensor processor architecture, eliminating the need for separate SIMD or SIMT engines. This merging reduces the number of distinct hardware components and simplifies the overall device architecture while maintaining scaling capability.
Solution Approach 2:
The patent extracts the scaling operation from the separate SIMD/SIMT engine domain and relocates it within the tensor processor. This extraction eliminates the dependency on separate scaling hardware, reducing system complexity.
3Productivity
If tile vector processors handle both multiplication and scaling, then productivity improves, but processing time increases due to workload saturation
Solution Approach 1:
The patent merges multiplication and scaling operations within the same tensor processor, allowing these operations to execute in parallel without requiring sequential processing. This eliminates the workload saturation issue that would occur if a single processor type had to handle both operations sequentially.
Data Source
AI summary
A hardware accelerator including input memory that receives first and second input matrices. The hardware accelerator further includes processing circuitry including one or more tiles that each include a respective tensor processor configured to receive a first and second input block of the first and second input matrices. Each tile receives a first block scale factor associated with rows of the first input block and a second block scale factor associated with columns of the second input block. Each tile multiplies the first input block by the second input block, applies the first block scale factor to rows of the result block, and applies the second block scale factor to columns of the result block to obtain a scaled result block. The processing circuitry further includes an accumulator that accumulates scaled result blocks to obtain a scaled result matrix, and output memory that receives and output the scaled result matrix.