Asynchronous Mixed Precision Update for Resistive Processing Unit Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mixed-precision dynamic neural network (DNN) training processes face high runtime and storage costs due to high-precision digital calculations, and existing low-precision approaches suffer from performance issues related to synchronous performance requirements.
Innovation Solution
Perform update calculations in low-precision digital, asynchronously with respect to other update calculations, reducing memory and runtime requirements by converting outputs from high precision to lower precision formats and utilizing dedicated digital processing units for update parameter calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision digital calculations are used for update parameters, then calculation accuracy is improved, but runtime cost and storage cost increase
Solution Approach 1:
The patent changes the precision parameter of digital calculations from high precision to low precision for update parameter calculations. This parameter change reduces the computational complexity and storage requirements while maintaining sufficient accuracy for the training process, directly resolving the contradiction between calculation accuracy and runtime cost.
2Loss of time
If low-precision calculations are used for update parameters, then runtime cost and storage cost are reduced, but performance issues arise due to synchronous performance requirements
Solution Approach 1:
The patent introduces dynamic asynchronous execution for update parameter calculations. Instead of rigid synchronous operations, the system allows update calculations to proceed asynchronously with other training operations, dynamically adjusting execution timing to avoid performance bottlenecks while maintaining low-precision efficiency.
Solution Approach 2:
The patent segments the training process into independent components that can execute asynchronously. Update parameter calculations are separated from the main training loop, allowing them to be computed independently in low precision without blocking other operations, thus resolving the performance issues associated with synchronous execution.
3Ease of operation
If synchronous update calculations are performed, then coordination between operations is simplified, but memory requirements and runtime increase
Solution Approach 1:
The patent transitions from static synchronous coordination to dynamic asynchronous coordination. Update calculations are decoupled from the main training flow and can execute independently, reducing the need for synchronized memory access and reducing overall memory requirements while maintaining operational coordination through asynchronous communication mechanisms.
Data Source
AI summary
A computer-implemented method, computer program product, and/or computer system that performs the following operations: (i) receiving outputs pertaining to a first step of a training process being performed on an analog resistive processing unit (RPU) array, the analog RPU array corresponding to a layer of a deep neural network (DNN); (ii) converting the outputs into a format having less precision, yielding converted outputs; (iii) initiating a calculation of an update parameter for a first step update pass of the layer utilizing the converted outputs; and (v) based, at least in part, on receiving outputs pertaining to a second step of the training process being performed on the analog RPU array, applying the update parameter for the first step update pass of the layer to the analog RPU array.