Fixed-Point Decimal Offset Correction for ML Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for adjusting the decimal point position of fixed-point number data in arithmetic processing apparatuses for machine learning models, based on statistical information, often lead to gaps between estimated and actual distributions, resulting in increased quantization errors and unstable learning processes.
Innovation Solution
An arithmetic processing apparatus that stores and processes error data related to the decimal point position of fixed-point number data, using statistical information to determine an offset amount for correcting the decimal point position, thereby improving the accuracy of the learning process by adjusting the decimal point position based on the tendency of errors in each iteration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the decimal point position is adjusted based on statistical information from a single iteration, then the circuit scale and power consumption are reduced, but gaps between estimated and actual distributions increase causing quantization errors
Solution Approach 1:
The patent performs preliminary actions by collecting statistical information across multiple iterations before making the final decimal point position adjustment. This preliminary data collection phase allows the system to build a more accurate picture of the actual data distribution, reducing gaps between estimated and actual distributions and thereby minimizing quantization errors while maintaining learning stability
Solution Approach 2:
The patent implements feedback by using the actual distribution information obtained from multiple iterations to correct and refine the decimal point position estimation. The system continuously compares estimated positions with actual distribution patterns and adjusts accordingly, creating a closed-loop control mechanism that reduces quantization errors and stabilizes the learning process
2Device complexity
If the decimal point position is adjusted based on statistical information, then the circuit scale is reduced, but quantization errors increase due to saturation or rounding
Solution Approach 1:
The system performs preliminary analysis of data distributions across multiple iterations before finalizing the decimal point position. This advance preparation allows for more accurate estimation that accounts for actual data patterns, reducing the need for excessive precision in the fixed-point representation and thereby reducing quantization errors while maintaining simplified circuit architecture
Solution Approach 2:
The patent dynamically changes the decimal point position parameter based on observed data distributions rather than using a fixed position. This adaptive parameter adjustment optimizes the balance between precision and circuit complexity by positioning the decimal point where it provides maximum benefit for the actual data being processed, minimizing quantization errors without requiring increased circuit scale
3Use of energy by moving object
If the decimal point position is adjusted based on statistical information, then power consumption is reduced, but learning accuracy decreases due to distribution gaps
Solution Approach 1:
The system performs preliminary data collection and distribution analysis across multiple iterations before making decimal point position adjustments. This preliminary phase enables more accurate positioning that better reflects actual data characteristics, reducing learning accuracy degradation while maintaining the power consumption benefits of fixed-point arithmetic
Solution Approach 2:
The patent implements feedback mechanisms where the system monitors actual distribution patterns and uses this information to refine decimal point position estimates. This feedback loop reduces gaps between estimated and actual distributions, thereby improving learning accuracy while preserving the energy efficiency gains from using fixed-point number systems
Data Source
AI summary
An arithmetic processing apparatus includes: a memory that stores, when a training of a given machine learning model is repeatedly performed in a plurality of iterations, an error of a decimal point position of each of a plurality of fixed-point number data obtained one in each of the plurality of iterations, the error being obtained based on statistical information related to a distribution of leftmost set bit positions for positive number and leftmost unset bit positions for negative number or a distribution of rightmost set bit positions of the plurality of fixed-point number data; and a processor coupled to the memory, the processor being configured to: determine, based on a tendency of the error in each of the plurality of iterations, an offset amount for correcting a decimal point position of fixed-point number data used in the training.