Adaptive Numerical Quantization for Large-Scale Tensor Computations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computational devices face inefficiencies in memory resources and computational logic due to the strain of performing high-precision matrix computations, particularly in large-scale operations like tensor computations in deep neural networks, which can be mitigated by reducing numerical precision without significant performance degradation.
Innovation Solution
A computational method and device that uses a reduced precision format by extracting a subset of bits from input numbers using a quantization factor to optimize the range of output numbers, dynamically adjusting to prevent overflow and underflow, and applying stochastic rounding for improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision floating-point format (32-bit) is used to maintain computational precision, then numerical accuracy is improved, but memory resources and computational logic are strained
Solution Approach 1:
The patent changes the precision parameter from 32-bit floating-point to variable-precision fixed-point formats (8-bit, 16-bit, or 32-bit), allowing the system to adaptively select the appropriate precision level based on computational requirements, thereby reducing memory strain while maintaining necessary accuracy
Solution Approach 2:
The system dynamically adjusts the precision level during computation by selecting different fixed-point formats based on the specific computational context, enabling flexible resource allocation that balances precision requirements with memory and computational constraints
2Device complexity
If numerical precision is reduced to alleviate memory and computational strain, then device complexity is reduced, but computational accuracy deteriorates
Solution Approach 1:
The patent segments the computational process into multiple stages, applying different precision levels to different stages: lower precision (8-bit or 16-bit) for intermediate computations where accuracy requirements are relaxed, and higher precision (32-bit) for final results where accuracy is critical, thereby maintaining overall computational accuracy while reducing overall system complexity
Solution Approach 2:
Different precision levels are applied locally to different parts of the computation based on their specific accuracy requirements, rather than uniformly applying high precision throughout, allowing the system to maintain accuracy only where necessary while reducing memory and computational strain in other areas
3Ease of manufacture
If fixed precision format is used for all computations, then implementation is simplified, but efficiency in large-scale repeated computations deteriorates
Solution Approach 1:
The system implements dynamic precision selection that adapts to the specific computational task at hand, automatically choosing the appropriate fixed-point format (8-bit, 16-bit, or 32-bit) based on the data range and precision requirements of each computation, thereby optimizing computational efficiency for large-scale repeated operations while maintaining implementation feasibility
Data Source
AI summary
A computational device includes an input memory, which receives a first array of input numbers having a first precision represented by N bits. An output memory stores a second array of output numbers having a second precision represented by M bits, M<N. Quantization logic reads the input numbers from the input memory, extracts from each input number a set of M bits, at a bit offset within the input number that is indicated by a quantization factor, and writes a corresponding output number based on the extracted set of bits to the second array in the output memory. A quantization controller sets the quantization factor so as to optimally fit an available range of the output numbers in the second array to an actual range of the input numbers in the first array in extraction of the M bits from the input numbers.