Dynamic Arithmetic Data Type Switching for Deep Learning Processors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current processors optimized for deep learning and machine learning face challenges in improving learning efficiency due to fixed arithmetic operation accuracy and inability to adapt to changes in arithmetic operation element values during the learning process, leading to potential stagnation and decreased efficiency.
Innovation Solution
An information processing apparatus that dynamically switches between 32-bit floating point and 8-bit fixed point arithmetic data types based on expressible ratios and fluctuation thresholds, automatically determining the appropriate data type for each arithmetic operation to maintain accurate and efficient learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If a processor uses fixed arithmetic operation accuracy (e.g., 8-bit fixed point or 16-bit floating point) optimized for deep learning, then electric power performance is improved, but the ability to cope with changes in arithmetic operation element values during learning deteriorates
Solution Approach 1:
The processor dynamically switches between 32-bit floating point and 8-bit fixed point arithmetic data types based on the learning stage. This dynamic adaptation allows the system to use high-precision 32-bit floating point arithmetic during early learning stages when value variations are large, and switch to energy-efficient 8-bit fixed point arithmetic during later stages when value variations decrease, thereby resolving the contradiction between power efficiency and adaptability
Solution Approach 2:
The system changes the arithmetic operation precision parameter based on the learning progress. By monitoring whether the variation in arithmetic operation element values exceeds a threshold, the system adjusts the data type parameter (32-bit floating point vs. 8-bit fixed point) to match the current learning stage requirements, optimizing both power consumption and learning effectiveness
2Use of energy by moving object
If arithmetic operation accuracy is deteriorated to improve electric power performance, then power efficiency is improved, but learning efficiency deteriorates
Solution Approach 1:
The system dynamically adjusts arithmetic precision based on learning stage requirements. During early learning stages when high precision is needed for effective learning, the system uses 32-bit floating point arithmetic. During later stages when value variations are small, it switches to 8-bit fixed point arithmetic, thereby maintaining learning efficiency while improving power efficiency
Solution Approach 2:
The system performs preliminary learning using 32-bit floating point arithmetic to establish a foundation where value variations become small. This preliminary high-precision learning enables subsequent switching to low-precision 8-bit fixed point arithmetic without significantly deteriorating overall learning efficiency, while achieving power efficiency improvements
3Loss of energy
If the arithmetic operation is switched from 32-bit floating point to 8-bit fixed point, then the number of arithmetic operations is reduced and power efficiency is improved, but the ability to cope with value changes during learning deteriorates
Solution Approach 1:
The system continuously monitors the variation in arithmetic operation element values during learning and uses this feedback to determine when to switch between 32-bit floating point and 8-bit fixed point arithmetic. When variation exceeds a threshold, the system switches to 32-bit floating point to maintain learning effectiveness; when variation is within the threshold, it switches to 8-bit fixed point to improve power efficiency
Solution Approach 2:
The system changes the arithmetic data type parameter based on the monitored value variation. By adjusting this parameter dynamically, the system optimizes the balance between power efficiency and adaptability to value changes during different learning stages
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An information processing apparatus includes: an arithmetic operation unit configured to perform an arithmetic operation using an arithmetic operation target and repeat the arithmetic operation by using a calculated arithmetic operation result; a ratio calculation unit configured to obtain a ratio of the number of elements, of the number of elements included in the arithmetic operation result, included in an expressible range as a predetermined-bit fixed point; and an arithmetic operation control unit configured to make the arithmetic operation unit perform the arithmetic operation by using the predetermined-bit fixed point based on the ratio calculated by the ratio calculation unit.