Segmented Neural Network Training Reducing Memory Usage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training large neural networks requires significant memory usage due to the need to store activations across multiple layers, which can lead to increased memory requirements and slower convergence during the training phase, especially when pipelining is involved.
Innovation Solution
Segmenting the neural network into multiple sections allows error propagation to be coupled directly to intermediate outputs, reducing the number of activations that need to be stored and enabling faster convergence by adjusting weights independently in each section.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the neural network uses pipelining to process data through multiple layers, then processing throughput is improved, but memory size increases significantly (quadratically)
Solution Approach 1:
The patent segments the neural network into multiple stages, where each stage processes a subset of layers. Activations are stored only for the current stage's layers rather than the entire network depth. This segmentation allows pipelining to continue across stages while reducing the memory burden at each individual stage, breaking the quadratic memory growth pattern.
Solution Approach 2:
The patent introduces a temporal dimension to memory management by using buffered storage that allows activations to be released after they have been used for backpropagation in previous stages. This temporal buffering approach enables the system to maintain pipelining throughput while reducing peak memory requirements by reusing memory resources across different time steps.
2Measurement precision
If the network depth is increased to improve model capacity, then recognition accuracy is improved, but memory usage increases
Solution Approach 1:
By dividing the deep network into multiple trainable stages that are trained independently and then combined, the patent enables training of very deep networks without requiring storage of activations for the entire depth. Each stage can be trained with moderate memory requirements, and the staged approach allows the overall network to achieve the depth necessary for high recognition accuracy.
Solution Approach 2:
The patent performs preliminary training of individual stages before combining them into the full network. This staged training approach allows each component to be optimized independently with manageable memory requirements, and the preliminary training results can be combined to form the complete deep network that achieves high accuracy without requiring all activations to be stored simultaneously.
3Measurement precision
If traditional backpropagation is used to train the entire network, then convergence accuracy is achieved, but training time increases due to storing all activations
Solution Approach 1:
The patent segments the backpropagation process into multiple passes, where each pass trains one stage of the network. This staged backpropagation allows gradients to be computed and applied to each stage independently, eliminating the need to store all activations from the entire network depth. The segmented approach achieves the same convergence accuracy as traditional backpropagation but significantly reduces training time by avoiding the storage bottleneck.
Data Source
AI summary
Embodiments of the present disclosure include systems and methods for training neural networks. In one embodiment, neural network may receive input data and produce output results in response to the input data and weights of the neural network. An error is determined at an output of the neural network based on the output results. The error is propagated in a reverse direction through the neural network from the output and one or more intermediate outputs to adjust the weights.