Outer Product Multiplication Apparatus with Internal Accumulation Paths
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The chunking accumulation method for AI processors leads to decreased speed of training and inference due to frequent memory access and expands hardware area, as it requires separate operation units and external memory storage for accumulation results.
Innovation Solution
An apparatus and method that utilize internal calculators to perform Multiply-Accumulate operations and generate chunking accumulation values without external memory storage, using accumulation data transmission paths to transfer intermediate values between calculators, allowing for continuous operation and minimizing hardware expansion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If chunking accumulation method is applied to perform accumulation operations on divided chunks, then information loss by quantization is reduced and accuracy is improved, but frequent memory access is required which decreases the speed of training and inference
Solution Approach 1:
The patent divides the accumulation operation into two stages: first stage performs accumulation on divided chunks using first internal calculators, second stage accumulates the results using second internal calculators. This segmentation allows continuous operation without frequent memory access while maintaining accuracy.
Solution Approach 2:
The patent introduces intermediate accumulation values as a mediator between the first internal calculators and second internal calculators. These intermediate values are transferred through accumulation data transmission paths, enabling continuous processing without external memory access and resolving the speed-accuracy tradeoff.
2Measurement precision
If chunking accumulation method is applied with separate operation units for two stages, then accumulation accuracy is maintained, but hardware area is expanded
Solution Approach 1:
The patent makes internal calculators universal by enabling them to function in different roles. First internal calculators perform first-stage accumulation, while second internal calculators perform second-stage accumulation. The same calculator units can be dynamically assigned to different stages, reducing overall hardware area while maintaining accuracy.
Solution Approach 2:
The patent reorganizes the calculator architecture from a single-dimension sequential processing to a two-dimension parallel structure with first and second internal calculators operating simultaneously on different chunks. This dimensional change allows continuous operation without expanding hardware area.
3Adaptability or versatility
If external memory is used to store accumulation results between chunks, then accumulation operation can be performed on divided chunks, but frequent memory access delays the operation and decreases processing speed
Solution Approach 1:
The patent merges the first internal calculators and second internal calculators into a unified calculator system with direct data transmission paths. This eliminates the need for external memory storage between chunks, allowing continuous accumulation operations without memory access delays while maintaining chunking capability.
Data Source
AI summary
Disclosed herein are an apparatus and method for a multiplication operation based on an outer product. The apparatus may include first internal calculators, each of which generates an intermediate accumulation value by performing a Multiply-Accumulate (MAC) operation, second internal calculators, each of which generates a chunking accumulation value using the intermediate accumulation value, and accumulation data transmission paths for enabling the output of any one of the first internal calculators to be input to any one of the second internal calculators.