Tile-Based Dynamic Quantization Without a Second Data Pass
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning (ML) model training and inference techniques require a second pass on data to calculate dynamic scales and offsets, leading to significant performance impact due to additional data retrieval and processing.
Innovation Solution
Implementing a compute unit that determines per tile scales within a single processing unit, allowing for in-place quantization operations without requiring a second pass, using techniques such as calculating mean, min/max, or historical scales to dynamically determine quantization scales on a per tile basis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If dynamic scales are calculated per tensor or per channel, then quantization accuracy is improved, but processing time increases due to requiring a second pass on the data
Solution Approach 1:
The patent segments the tensor data into multiple tiles that can be processed independently in parallel. Each tile has its own scale calculation that can be performed concurrently with other tiles, eliminating the need for a sequential second pass while maintaining per-tile quantization accuracy.
Solution Approach 2:
The patent calculates tile scales during the first pass through the data by maintaining running statistics (sum, min, max) as data flows through the compute unit. This preliminary calculation of scale information eliminates the need for a separate second pass, resolving the time penalty associated with dynamic quantization.
2Measurement precision
If all data values are known before quantization, then accurate dynamic scales can be calculated, but memory requirements increase to store all values
Solution Approach 1:
The patent divides the data into smaller tile segments that fit within the compute unit's memory. Each tile is processed independently with its own scale calculation, allowing accurate dynamic quantization without requiring the entire tensor to be stored in memory simultaneously.
Solution Approach 2:
The patent introduces intermediate statistics (running sum, min, max) that are maintained during the data pass through the compute unit. These intermediates serve as mediators that capture the essential information needed for scale calculation without requiring storage of all individual data values.
3Measurement precision
If a second pass is performed to calculate dynamic scales, then quantization precision is improved, but device complexity increases due to additional kernel operations
Solution Approach 1:
The patent merges the scale calculation operations with the existing data processing pipeline by integrating running statistics computation into the single pass through the compute unit. This consolidation eliminates the need for separate kernel operations while maintaining precise dynamic scale calculation for each tile.
Data Source
AI summary
Embodiments herein dynamically calculate a scale/offset on a per tile (or per block) basis rather than on a per tensor or channel basis. This enables the scale to be determined in place in the compute unit (e.g., a workgroup)—e.g., without having to perform a second pass or retrieve data from main memory. The scale for the tile can be determined by the compute unit using different techniques. In one embodiment, the scale is determine from the data in the tile itself. In another embodiment, a historical scale could be used.