Neural Network Training Memory Profiling for Low-Latency Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training of learning networks, such as deep neural networks, is time-consuming and memory-intensive, particularly for edge devices, often leading to memory exhaustion due to the storage of information like loss functions and activations.
Innovation Solution
A method involving profiling the model to determine an optimized training technique that reduces latency and memory usage by selectively loading activations from remote memory and employing techniques like backpropagation, activation recomputation, or feedback alignment, combined with compute-in-memory hardware modules for efficient vector-matrix multiplications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If standard training techniques (backpropagation) are used, then training accuracy can be achieved, but memory consumption becomes excessively high
Solution Approach 1:
The training process is segmented into distinct phases (forward pass, backward pass, weight update) with selective memory retention. Only essential intermediate values are stored, while others are recomputed when needed, dividing the memory burden into manageable segments across different training iterations.
Solution Approach 2:
Activations are pre-computed and stored in a compressed format during the forward pass, and gradient information is pre-calculated during the backward pass. This preliminary action reduces the need to store full-precision intermediate values throughout training, lowering overall memory consumption.
2Reliability
If standard training techniques are used, then model weights can be optimized, but training time becomes excessively long
Solution Approach 1:
The precision of intermediate value storage is changed from full-precision to reduced precision formats. This parameter change reduces computation time for memory operations while maintaining sufficient accuracy for effective weight optimization, balancing training speed with model performance.
3Measurement precision
If full precision intermediate values are stored, then computation accuracy is maintained, but memory footprint becomes unmanageably large
Solution Approach 1:
The precision parameter of stored intermediate values is changed from full precision to reduced precision formats. This allows the system to maintain adequate computation accuracy while dramatically reducing the memory footprint to manageable levels for edge device deployment.
Data Source
AI summary
A method is described. The method includes profiling a model for a learning network having a plurality of layers and associated memory. The layers include weight layers and activation layers. The plurality of weight layers including weights. The method also includes determining, based on the profiling, a training technique for the model on the learning network. The determination of the training technique includes optimizing a latency for at least one training iteration for a capacity of the associated memory. The training iteration(s) include at least one update of the weights for the weight layers.