Dynamic Decimal Point Adjustment for Fixed-Point Deep Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep training with fixed-point numbers often results in degraded precision due to infrequent adjustment of decimal point positions, leading to saturation or underflow issues, which can hinder convergence and accuracy.
Innovation Solution
An operation processing device that acquires statistical information on bit distributions in fixed-point data and updates the decimal point position dynamically, using a processor with specialized units for statistical information acquisition and aggregation to optimize fixed-point operations during deep training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If fixed-point numbers are used for deep training operations, then circuit size and power consumption are reduced, but precision is degraded due to narrow dynamic range
Solution Approach 1:
The patent implements dynamic fixed-point processing where the decimal point position is adjusted dynamically during deep training operations. The system reviews and updates the decimal point position at fixed timing based on statistical information about data distribution, allowing the fixed-point representation to adapt its dynamic range to match the actual data characteristics, thereby maintaining precision while using fixed-point arithmetic
Solution Approach 2:
The patent changes the parameter of decimal point position (Q format) based on statistical analysis of the data. By acquiring statistical information on bit distributions and adjusting the decimal point position accordingly, the system optimizes the fixed-point representation to maintain adequate precision for the current data range while preserving the area and power benefits of fixed-point arithmetic
2Measurement precision
If decimal point position is adjusted frequently to maintain precision, then operation precision is improved, but circuit complexity and processing overhead increase
Solution Approach 1:
The patent implements periodic adjustment of the decimal point position at fixed timing intervals during deep training operations. Rather than continuously adjusting the decimal point position, the system reviews and updates it periodically based on accumulated statistical information, reducing the complexity and overhead of decimal point management while maintaining adequate precision through timely adjustments
Solution Approach 2:
The system performs self-adjustment of the decimal point position by automatically acquiring statistical information on bit distributions and updating the Q format based on this information. This self-service mechanism eliminates the need for complex external control logic to manage precision, allowing the fixed-point processing unit to adapt its own parameters based on observed data characteristics
Data Source
Figure 1
Figure 2
Figure 3
AI summary
To reduce a circuit scale and power consumption by performing deep training with a fixed point number with high precision. An operation processing device includes: an acquisition circuit that acquires statistical information on distribution of bits in fixed point number data after execution of an instruction on the fixed point number data, and an update circuit that updates a decimal point position of the fixed point number data.