Block Data Quantization Using Shared Scalar Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing size of neural networks and the use of high-precision data types lead to costly storage and computational challenges, particularly in vector processors, necessitating efficient quantization methods to reduce bit-width and computational costs.
Innovation Solution
A scalar processor is integrated with a vector processor to predict and adjust shared scalars for quantization, using feedback mechanisms to minimize quantization error, allowing for high-performance vectorized operations with reduced memory requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision data types are used to maintain accuracy, then measurement precision is improved, but storage cost and computational cost increase
Solution Approach 1:
The patent segments data representation into two parts: a shared scalar value stored once for a block of data, and individual block elements stored with reduced precision. This segmentation allows the system to maintain high effective precision for all elements while storing only one full-precision reference value, dramatically reducing storage requirements compared to storing full-precision data for every element.
Solution Approach 2:
The patent uses a copy-based approach where a single shared scalar value is copied and applied to multiple block elements during computation. Instead of storing multiple full-precision values, the system stores one reference scalar and replicates its usage across the entire block, reducing storage cost while maintaining computational precision through the shared reference.
2Measurement precision
If high-precision data types are used to maintain accuracy, then measurement precision is improved, but computational cost increases
Solution Approach 1:
The computational process is segmented into operations on the shared scalar and operations on block elements. By separating the precision-critical scalar operations from the less precision-sensitive element operations, the system can use high-precision arithmetic only where necessary (for the shared scalar) and lower-precision arithmetic for bulk operations, reducing overall computational cost while maintaining result accuracy.
Solution Approach 2:
The shared scalar value is copied and applied to multiple block elements during computation rather than performing full-precision operations on each element individually. This copying approach allows the system to perform expensive high-precision operations once on the shared scalar, then propagate the results to multiple elements, significantly reducing the total number of expensive computational operations required.
3Productivity
If block size is increased to improve processing efficiency, then productivity is improved, but device complexity increases due to larger register requirements
Solution Approach 1:
The patent segments the block data into elements that can be processed with a shared scalar, allowing large blocks to be handled without requiring proportionally large registers. By segmenting the precision requirements (shared scalar vs. block elements), the system can process large blocks efficiently while keeping register requirements manageable, as only the shared scalar needs full-precision storage rather than every element in the block.
Data Source
AI summary
A scalar processor associated with a vector processor reduces the quantization error for blocked data with a relatively small register size by predicting adjustments for shared scalars used in runtime quantization. The scalar processor provides a recommended scale value to the vector processor for scaling a block of data from a wide data type format to a narrow data type format. The scalar processor and the vector processor share a register at which the scalar processor stores the recommended scale value and from which the vector processor accesses the recommended scale value. The vector processor performs an operation to quantize at least a portion of the block of data by applying a scale value that is based on the recommended scale value.