Weight Expansion Engine for Neural Network Data Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The performance of neural network hardware systems is limited by memory access latency, which restricts the achievable data throughput due to high memory data traffic, making it challenging to improve memory bandwidth without increasing power and chip area.
Innovation Solution
A computing system that includes a memory device storing compressed weights and a configurable weight expansion engine, which extracts and expands these weights to reduce memory data traffic, allowing the system to operate with increased arithmetic intensity and parallel computations, thereby improving performance beyond memory bandwidth limitations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If memory bandwidth is increased to improve data throughput, then data throughput is improved, but power consumption and chip area increase
Solution Approach 1:
The weight data is segmented into two parts: high-order bits and low-order bits. The high-order bits are stored in memory and transferred to the computing engine, while the low-order bits are generated locally through bit expansion circuits. This segmentation reduces memory bandwidth requirements by only transferring the essential high-order bits, thereby improving data throughput without proportionally increasing power consumption and chip area.
Solution Approach 2:
The patent extracts only the necessary high-order bits from the weight data for storage and transfer, while the less critical low-order bits are generated locally through bit expansion. This extraction approach reduces the volume of data that needs to be transferred over memory bandwidth, thus improving data throughput without requiring a proportional increase in memory bandwidth resources that would consume more power and chip area.
2Productivity
If memory bandwidth is increased to improve data throughput, then data throughput is improved, but chip area increases
Solution Approach 1:
The weight data is segmented into two parts: high-order bits and low-order bits. The high-order bits are stored in memory and transferred to the computing engine, while the low-order bits are generated locally through bit expansion circuits. This segmentation reduces memory bandwidth requirements by only transferring the essential high-order bits, thereby improving data throughput without proportionally increasing power consumption and chip area.
Solution Approach 2:
The patent extracts only the necessary high-order bits from the weight data for storage and transfer, while the less critical low-order bits are generated locally through bit expansion. This extraction approach reduces the volume of data that needs to be transferred over memory bandwidth, thus improving data throughput without requiring a proportional increase in memory bandwidth resources that would consume more power and chip area.
3Loss of time
If memory access latency is reduced by increasing memory bandwidth, then memory access latency is reduced, but power consumption increases
Solution Approach 1:
The weight data is segmented into two parts: high-order bits and low-order bits. The high-order bits are stored in memory and transferred to the computing engine, while the low-order bits are generated locally through bit expansion circuits. This segmentation reduces memory bandwidth requirements by only transferring the essential high-order bits, thereby improving data throughput without proportionally increasing power consumption and chip area.
Solution Approach 2:
The patent performs bit expansion of the high-order bits to generate low-order bits before the actual computation using preliminary action. The bit expansion circuits generate the low-order bits in advance, so that when the high-order bits are fetched from memory, the complete weight data can be reconstructed quickly without requiring additional memory access. This preliminary action reduces memory access latency without requiring increased memory bandwidth that would consume more power.
Data Source
AI summary
Disclosed herein are techniques for obtain weights for neural network computations. In one embodiment, an integrated circuit may include an arithmetic circuit configured to perform arithmetic operations for a neural network. The integrated circuit may also include a weight processing circuit configured to: acquire data from a memory device; receive configuration information indicating a size of each quantized weight of a set of quantized weights; extract the set of quantized weights from the data based on the size of the each weight indicated by the configuration information; perform de-quantization processing on the set of quantized weights to generate a set of de-quantized weights; and provide the set of de-quantized weights to the arithmetic circuit to enable the arithmetic circuit to perform the arithmetic operations. The memory device may be part of or external to the integrated circuit.