Dynamic Floating-Point Precision Control for ML Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing floating-point arithmetic operations often require higher precision than necessary, leading to slower and less memory-efficient processing, particularly in applications like machine learning, where lower precision operations can be sufficient and more efficient.
Innovation Solution
The system decomposes numbers from a higher precision floating-point format into a lower precision format with fewer significand bits, allowing for faster arithmetic operations by approximating results using a plurality of decomposed numbers, with the number of terms determined by accuracy demand, and storing results in the lower precision format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If higher precision floating-point format is used, then numerical accuracy is improved, but processing speed deteriorates
Solution Approach 1:
The system dynamically adjusts the precision of floating-point operations based on the specific computational context and accuracy requirements. Different operations can use different precision levels (e.g., BF16 for some operations, FP32 for others), allowing the system to optimize between speed and accuracy in real-time rather than being locked into a single precision mode throughout the entire computation pipeline.
Solution Approach 2:
The patent changes the precision parameter of floating-point operations from a fixed value to a variable that can be adjusted based on requirements. By using formats like BF16 (which has fewer significand bits than standard FP32), the system can perform faster operations when high precision is not strictly necessary, while still maintaining FP32 capability when accuracy is critical.
2Measurement precision
If higher precision floating-point format is used, then numerical accuracy is improved, but memory efficiency deteriorates
Solution Approach 1:
The system changes the precision parameter to reduce memory footprint. By using lower precision formats like BF16 for storing intermediate results and activations in neural network computations, the system reduces memory consumption by a factor of 2-4x compared to using FP32 throughout, while still achieving acceptable accuracy through selective use of higher precision where needed.
3Speed
If lower precision floating-point format is used, then processing speed is improved, but numerical accuracy deteriorates
Solution Approach 1:
The patent segments the computational pipeline into different stages with different precision requirements. For example, weight storage and certain intermediate computations may use lower precision (BF16) for speed, while activation functions and final output computations use higher precision (FP32) for accuracy. This segmentation allows the system to achieve overall speed improvements without sacrificing critical accuracy in key computational stages.
Solution Approach 2:
Different parts of the computational system use different precision levels based on their specific requirements. The system applies local quality by allowing BF16 format in regions where speed is prioritized (such as matrix multiplication layers) while maintaining FP32 in regions where accuracy is critical (such as softmax operations or loss computation), rather than applying a uniform precision level across the entire system.
Data Source
AI summary
Methods and apparatuses include a processing unit which helps control the speed and computational resources required for arithmetic operations of two numbers in a first format. The control unit of the processing unit approximates the arithmetic operations using a plurality of decomposed numbers in a second format that facilitates faster calculations than the first format, such that performing arithmetic operations using the decomposed numbers is capable of approximating the results of the arithmetic operations of the two numbers in the first format.