Neural Network Training Memory Optimization via Selective Intermediate Output Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network training operations face inefficiencies due to memory constraints, leading to increased training time and resource underutilization, especially when dealing with large datasets and complex neural networks, as existing methods require storing intermediate outputs for all layers, which limits the batch size and computation efficiency.
Innovation Solution
A training module manages memory and computation resources by selectively storing intermediate outputs of only certain neural network layers, allowing for larger input datasets and reducing the number of batches, using heuristics such as layer complexity and computation resources to optimize storage and regeneration of intermediate outputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If intermediate outputs of all neural network layers are stored in memory, then training accuracy is maintained, but memory resources are excessively consumed
Solution Approach 1:
The patent extracts only the necessary intermediate outputs from the neural network layers for storage in memory, rather than storing all intermediate outputs. By identifying and storing only critical intermediate results that are essential for backpropagation and gradient calculation, the system maintains training accuracy while significantly reducing memory consumption.
Solution Approach 2:
The patent applies local quality by differentiating which intermediate outputs require storage based on their specific importance to the training process. Different layers and different intermediate outputs are treated differently - some are stored while others are discarded, creating a non-uniform storage strategy that optimizes memory usage while preserving necessary information for accurate gradient computation.
2Productivity
If batch size is increased to improve training efficiency, then productivity increases, but memory constraints are violated
Solution Approach 1:
The patent changes the parameter of batch size by enabling larger batch sizes than traditionally possible given the memory constraints. By selectively storing intermediate outputs, the system can accommodate larger batches in memory without exceeding available resources, thereby increasing the number of samples processed per iteration and improving overall training efficiency.
3Reliability
If all intermediate outputs are retained for backpropagation, then computation accuracy is maintained, but training time increases due to repeated computations
Solution Approach 1:
The patent applies preliminary action by pre-storing selected intermediate outputs in memory during the forward pass, making them readily available for the backpropagation phase. This eliminates the need to recompute these intermediate values during backward propagation, significantly reducing training time while maintaining computation accuracy through selective retention of critical intermediate results.
4Reliability
If memory allocation is increased to store more intermediate outputs, then training completeness improves, but resource utilization becomes inefficient
Solution Approach 1:
The patent applies partial action by storing only the necessary portion of intermediate outputs - enough to maintain training completeness but not excessive amounts. This selective storage approach ensures that all critical information needed for accurate backpropagation is preserved while avoiding waste of memory resources on redundant or non-essential intermediate values.
Data Source
AI summary
Methods and systems for training a neural network are provided. In one example, an apparatus comprises a memory that stores instructions; and a hardware processor configured to execute the instructions to: control a neural network processor to perform a loss gradient operation to generate data gradients; after the loss gradient operation completes, control the neural network processor to perform a forward propagation operation to generate intermediate outputs; control the neural network processor to perform a backward propagation operation based on the data gradients and the intermediate outputs to generate weight gradients; receive the weight gradients from the neural network processor; and update weights of a neural network based on the weight gradients.