Neural Network Weight Block Compression for Low-Latency Matrix Multiply
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based neural network models require significant computational resources and memory, leading to inefficiencies in processing due to high latency in memory operations, especially when handling large data sizes that exceed on-chip memory capacity.
Innovation Solution
A matrix multiply compute apparatus and method that employs data compression and decompression techniques using block floating-point formats, allowing for efficient storage and processing of weight matrix elements by converting them into a format that reduces memory requirements without needing separate hardware pathways, utilizing a crossbar device and register devices to determine max exponents and converted mantissa blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If weight matrix elements are stored in full precision format, then computational accuracy is maintained, but memory storage requirements and access latency increase significantly
Solution Approach 1:
The patent segments weight matrix elements into two distinct components: mantissa blocks (storing significant digits) and scale factors (storing exponent information). This segmentation allows the system to store only the most critical precision information in the mantissa while using scale factors to represent magnitude, thereby reducing overall storage requirements and improving memory access speed while maintaining computational accuracy.
Solution Approach 2:
The patent changes the storage format parameter from full precision floating-point representation to a compressed format with separate mantissa and scale factor fields. By adjusting the allocation of bits between mantissa and exponent components, the system optimizes the balance between storage efficiency and computational precision, reducing memory latency without sacrificing accuracy.
2Quantity of substance
If data is compressed into block floating-point format, then memory storage capacity is reduced, but additional conversion processing is required
Solution Approach 1:
The patent performs preliminary conversion of weight matrix elements into block floating-point format with separate mantissa and scale factor components during the weight loading phase. This preliminary action ensures that data is stored in the optimized compressed format from the beginning, avoiding the need for complex real-time conversion during computation, thereby reducing overall processing complexity.
Solution Approach 2:
The patent introduces scale factors as intermediary components that mediate between the compressed mantissa blocks and the actual weight values. These scale factors serve as a simple lookup mechanism that restores the full precision information when needed, eliminating the need for complex decompression algorithms and reducing conversion processing complexity.
3Speed
If separate hardware pathways are implemented for compressed data processing, then processing speed improves, but silicon area costs increase
Solution Approach 1:
The patent designs the compute apparatus to handle block floating-point data using the same existing hardware pathways that process standard floating-point data. The universal design allows existing ALUs, registers, and data buses to operate with compressed weight matrices without requiring dedicated separate hardware, thereby maintaining processing speed while avoiding additional silicon area costs.
Solution Approach 2:
The patent enables the existing compute apparatus to self-serve compressed block floating-point data by leveraging its inherent ability to handle floating-point operations. The hardware naturally accommodates the compressed format's mantissa and scale factor structure without requiring specialized processing units, thus achieving fast processing speeds using the same silicon real estate.
Data Source
AI summary
A matrix multiply compute apparatus and method of operation therefor. The apparatus includes a memory configured to store weight matrix elements in a first format including columns of scale factors and mantissa blocks. A first register is configured to receive the scale factors of each weight matrix column via a crossbar coupled to the memory, the converter is configured to determine max exponents using the scale factors of the columns, and a second register is configured to store the max exponents. Also, the first register is configured to receive the mantissa blocks of the columns, and the converter is configured to determine converted blocks using all of the scale factors and the mantissa blocks. A weight buffer receives the converted blocks and max exponents as the elements in a second format that is used by a compute device to determine matrix multiply outputs, which are stored in an output buffer.