Broadcast-Data TPU MAC Architecture for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing tensor processing unit (TPU) architectures face inefficiencies in processing latency, data exchange, and filter matrix storage due to their multi-data approaches, which lead to increased latency and complexity in tensor operations.
Innovation Solution
The proposed broadcast-data TPU architecture allows simultaneous execution of multiply-accumulate operations across all MAC processors using a shared input data operand, reducing processing latency and eliminating cycle-to-cycle data exchange, with each MAC processor executing multiple operations independently and storing weighting operands in a non-skewed manner, enabling flexible tensor mapping and aggregation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multi-data approach is used in TPU architecture, then parallel processing capability is improved, but processing latency increases
Solution Approach 1:
The patent merges multiple data processing streams into a single shared input data operand that is broadcast to all MAC processors simultaneously. This allows all processors to operate in parallel on the same data with different filter weights, achieving high parallel processing capability while eliminating the latency associated with sequential data delivery in multi-data approaches.
Solution Approach 2:
The patent segments the filter matrix into multiple filter weight operands, with each MAC processor receiving a different filter weight from the same shared input data. This segmentation of the weight dimension rather than the data dimension enables parallel processing without increasing data delivery latency.
2Productivity
If multi-data approach is used in TPU architecture, then processing throughput is improved, but data exchange complexity increases
Solution Approach 1:
The patent extracts the data exchange complexity from the system by using a broadcast mechanism where a single shared input data operand is delivered once to all MAC processors simultaneously. This eliminates the need for complex cycle-to-cycle data exchange between processors that characterizes multi-data approaches, while maintaining high processing throughput through parallel weight multiplication.
3Productivity
If skewed storage of filter weights is used, then memory access efficiency is improved, but aggregation complexity increases
Solution Approach 1:
The patent uses homogeneous (non-skewed) storage of filter weights in the filter weight memory, where weights are stored in their natural matrix order without reordering. This simplifies the aggregation process because results from multiple MAC processors can be directly summed without complex indexing or reordering operations, while memory access efficiency is maintained through the broadcast architecture that delivers the same input data to all processors simultaneously.
Data Source
AI summary
An integrated circuit device includes one or more broadcast data paths, a weighting-value memory and multiply-accumulate (MAC) units. The MAC units are coupled in common to each of the broadcast data paths and coupled to receive respective weighting values from the weighting-value memory via respective weighting-value paths. Each of the MAC units includes MAC circuits that each receive an input data value via a respective one of the broadcast data paths and a shared one of the weighting values via a shared one of the respective weighting-value paths; generate a sequence of multiplication products by multiplying the input data value with the shared one of the weighting values; and accumulate a sum of the multiplication products. A configuration value stored within a programmable register controls the number of timing cycles over which the sum of the multiplication products is accumulated.