Tensor Processing Circuitry With Shared Floating-Point Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional general-purpose processors like CPUs and GPUs face performance bottlenecks and excessive power consumption due to their lack of specialized architecture for handling the vast parallelism and dataflow inherent in tensor computations, particularly in machine learning applications.
Innovation Solution
The introduction of tensor processing circuitry with unnormalized floating-point intermediaries that amortize the hardware required for normalization across multiple dot product units, using an unnormalized accumulator and storing outputs in an unnormalized format, along with overflow detection and partial normalization to enhance efficiency and reduce processing costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If normalization circuitry is included in each dot product unit, then normalization accuracy is improved, but hardware complexity and power consumption increase
Solution Approach 1:
The patent merges the normalization function from individual dot product units into a shared normalization circuitry that serves multiple dot product units. Instead of each dot product unit having its own normalization circuit, the normalization operations are consolidated into a separate circuit that processes outputs from multiple dot product units, thereby reducing overall hardware complexity while maintaining normalization accuracy.
Solution Approach 2:
The shared normalization circuitry is designed to handle normalization tasks for multiple dot product units universally. This single circuit performs the normalization function that would otherwise require separate dedicated circuits in each dot product unit, making the normalization hardware multi-functional and reducing redundancy in the system.
2Measurement precision
If normalization is performed after every multiply-accumulate operation, then computational accuracy is improved, but processing speed decreases
Solution Approach 1:
The patent performs preliminary accumulation of unnormalized results in the second-stage circuitry before applying normalization. Multiple multiply-accumulate operations can proceed and accumulate their results without interruption for normalization. The normalization is applied in advance to the accumulated result rather than after each individual operation, maintaining accuracy while improving processing speed by allowing continuous computation.
Solution Approach 2:
The system maintains continuous computation by allowing the first-stage dot product units to continuously perform multiply-accumulate operations and pass results to the second-stage circuitry. The normalization circuitry operates continuously on the accumulated unnormalized results without interrupting the computation flow, ensuring that useful computational actions proceed without interruption while still achieving proper normalization.
3Productivity
If unnormalized format is used for accumulation, then processing efficiency is improved, but overflow risk increases
Solution Approach 1:
The patent introduces an intermediary unnormalized format that serves as a temporary storage state between the multiply-accumulate operations and the final normalized output. The second-stage circuitry accumulates results in this unnormalized format, which allows more efficient processing and larger accumulation ranges. The unnormalized format acts as a mediator that enables efficient computation while the normalization circuitry converts it to the proper normalized format for final output, managing overflow risk through the conversion process.
Data Source
AI summary
A tensor processing circuitry comprising a plurality of dot product units and normalization circuitry. Each dot product unit comprises first-stage circuitry and second-stage circuitry. The first-stage circuitry is configured to receive a plurality of input values and perform at least a multiply-accumulate operation on pairs of the plurality of input values, the multiply-accumulate operation produces an output value in a unnormalized floating-point format. The second stage circuitry is configured to receive a plurality of the unnormalized floating-point output values from the first stage circuitry and perform an accumulate operation on each of the received unnormalized floating-point output values to generate an unnormalized result. The unnormalized result of the accumulate operation is then output to the normalization circuitry which normalizes the unnormalized results.