Deep Learning Precision Switching for Failure-Aware Recalculation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning techniques using 8-bit or 16-bit operations are prone to failures due to inefficient detection of learning anomalies, leading to wasteful recalculations and inability to switch back to higher precision operations effectively, resulting in reduced calculation efficiency and performance.
Innovation Solution
A computer-readable recording medium and information processing apparatus that detects signs of learning failures in low-bit operations, rolls back to higher precision operations for recalculations, and allows switching back to low-bit operations when conditions permit, optimizing arithmetic precision in deep learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If 8-bit or 16-bit operations are used in machine learning, then calculation speed is improved and power performance is enhanced, but the system becomes more prone to learning failures and requires wasteful recalculations
Solution Approach 1:
The patent implements dynamic switching between operation precisions (32-bit floating-point and 8-bit/16-bit fixed-point) based on training progress and detected learning states. The system transitions from high-precision operations in the first half of training to low-precision operations in the second half, and can dynamically rollback when learning failures are detected, optimizing both speed and reliability throughout the training process.
Solution Approach 2:
The patent changes the precision parameter of numerical operations during training. It uses 32-bit floating-point operations initially, then switches to 8-bit or 16-bit fixed-point operations when appropriate, and can rollback to higher precision when failures occur. This parameter changing approach enables the system to achieve high calculation speed while maintaining learning reliability through adaptive precision management.
2Reliability
If training continues until learning failure is apparent, then learning completeness is ensured, but a number of wasteful calculations are performed
Solution Approach 1:
The patent performs preliminary detection of learning failures by monitoring specific indicators (such as loss value trends, gradient norms, or parameter changes) during training. When a failure sign is detected, the system proactively rolls back to a previous checkpoint before complete learning failure occurs, preventing wasteful continuation of failed training and reducing overall calculation time while maintaining learning effectiveness.
3Reliability
If switching to higher precision operations is performed for recalculations, then learning reliability is improved, but calculation efficiency is reduced
Solution Approach 1:
The patent implements dynamic precision switching that adapts to training needs. The system uses low-precision operations (8-bit/16-bit) for the majority of training iterations to maintain high calculation efficiency, and temporarily switches to high-precision operations (32-bit) only when learning failures are detected and rollback is necessary. This dynamic approach minimizes the time spent in lower-efficiency high-precision mode while ensuring learning reliability when needed.
4Quantity of substance
If 8-bit or 16-bit fixed-point operations are used, then memory resources are conserved, but the system is more likely to result in failed machine learning
Solution Approach 1:
The patent dynamically adjusts the precision of fixed-point operations during training. It starts with 32-bit floating-point operations that use more memory but ensure reliability, then transitions to 8-bit or 16-bit fixed-point operations that conserve memory resources. The system monitors learning progress and can rollback to higher precision representations when failures are detected, enabling efficient memory usage while maintaining learning success rate through adaptive precision management.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer is caused to perform processing of: detecting, in deep learning, a sign of a failure in learning in operations that are performed with a lower number of bits compared with operations that are performed with a certain number of bits; rolling back to an operation where the sign is detected and performing a recalculation by an operation with the certain number of bits; determining whether returning from operations with the certain number of bits to operations with the lower number of bits is allowed; and, when the returning to operations with the lower number of bits is allowed, switching to operations with the lower number of bits.