Fine-Grained Selective Quantization to Maximize Hardware Resource Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning (ML) hardware accelerators face inefficiencies in hardware resource utilization due to underutilization of resources like register files, scratch pad spaces, ALU units, and memory bandwidth during mixed-precision quantization, leading to suboptimal performance and accuracy in ML operations.
Innovation Solution
Implement fine-grained mixed precision quantization by partitioning high-level operations into sub-operations with different precision data formats and generating custom kernels to execute these sub-operations simultaneously, leveraging multiple ALU pipelines and optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If mixed-precision quantization is applied to ML operations, then hardware resource utilization improves, but accuracy of ML operations deteriorates
Solution Approach 1:
The patent segments ML operations into multiple sub-operations with different precision requirements. Critical sub-operations use higher precision (FP32) to maintain accuracy, while non-critical sub-operations use lower precision (INT8) to maximize hardware utilization. This segmentation allows simultaneous execution on different ALU pipelines, resolving the contradiction between resource utilization and accuracy.
Solution Approach 2:
The patent applies different data precision formats to different parts (sub-operations) of the ML operation based on their specific accuracy requirements. By assigning FP32 to accuracy-sensitive sub-operations and INT8 to less sensitive ones, the system maintains overall accuracy while maximizing hardware resource utilization across the entire operation.
2Productivity
If lower-precision data formats (INT8, INT4) are used, then memory bandwidth and compute throughput improve, but computational accuracy deteriorates
Solution Approach 1:
The patent divides ML operations into sub-operations that can be executed in parallel on different ALU pipelines. Some sub-operations use INT8 or INT4 for high throughput, while others use FP32 for high accuracy. The segmented approach allows the system to achieve overall high throughput while maintaining accuracy in critical computational paths.
Solution Approach 2:
The patent creates a composite computation approach by combining multiple precision formats (FP32, INT8, INT4) within a single ML operation. This composite strategy leverages the high throughput of low-precision formats and the high accuracy of high-precision formats, achieving a balance that neither format could achieve alone.
3Measurement precision
If high-precision floating point (FP32) computation is used, then accuracy is maintained, but hardware resource utilization and throughput deteriorate
Solution Approach 1:
The patent segments the computation workload to execute only the necessary portion in high-precision FP32 on dedicated pipelines, while routing other portions to low-precision INT8/INT4 pipelines for parallel execution. This segmentation maintains accuracy where needed while maximizing overall throughput through parallel low-precision computation.
Solution Approach 2:
The patent applies high-precision FP32 computation partially, only to sub-operations that require it for accuracy. Non-critical sub-operations use lower precision, avoiding the throughput penalty of full FP32 computation while maintaining necessary accuracy in critical paths.
4Device complexity
If register file size is increased to store more variables, then operational complexity increases, but hardware resource utilization deteriorates
Solution Approach 1:
The patent segments variables into groups based on their precision requirements and stores them in different register files or memory spaces. This segmentation allows efficient utilization of available register space without requiring a large increase in total register file size, as each segment uses only the precision appropriate for its variables.
Solution Approach 2:
The patent changes the precision parameter of variables dynamically based on their computational requirements. By using lower precision (INT8/INT4) for non-critical variables, the system reduces the memory space required in register files, allowing more variables to be stored without proportionally increasing hardware resource utilization.
Data Source
AI summary
Techniques for performing fine-grained mixed precision quantization for an ML model are disclosed. A quantizable operation is identified. This quantizable operation is partitioned into multiple sub-operations having multiple different precision data formats. One or more kernels are generated. These kernel(s) are tasked with simultaneously executing the sub-operations. Consequently, the first sub-operation, which has the first precision data format, is executed simultaneously with the second sub-operation, which has the second, different precision data format.