Adaptive Activation-Gradient Sparsification for Lower-Memory DNN Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deep neural network (DNN) training methods face challenges with high computation and memory demands, and gradient sparsification techniques fail to effectively reduce memory requirements while causing hardware overhead and decreasing training accuracy.
Innovation Solution
A neural network operation method involving sparsification that includes receiving an activation gradient and a threshold, updating the threshold based on the gradient, and performing operations using the updated threshold to generate sparse data, reducing computation and memory needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional gradient sparsification technique (top-k) is used, then computation amount is reduced, but memory demand is not reduced and additional preprocessing time is required
Solution Approach 1:
The patent applies preliminary action by performing sparsification on activation gradients during the forward propagation phase before backpropagation begins. This allows the sparsity pattern to be established in advance, enabling the hardware to skip unnecessary memory accesses and computations during the backward pass without requiring additional preprocessing time.
Solution Approach 2:
The patent extracts only the significant activation gradients above a certain threshold and sets the rest to zero, creating a sparse representation. This extraction principle reduces both computation amount and memory demand by eliminating redundant gradient values that would otherwise require storage and processing during backpropagation.
2Productivity
If conventional sparse matrix multiplication architectures are used, then computation amount is reduced, but hardware overhead increases due to irregularity of elements
Solution Approach 1:
The patent applies local quality by making the sparsity pattern adaptive to local characteristics of the neural network layer. Different layers or even different positions within a layer can have different sparsity thresholds, allowing the hardware architecture to optimize for local density patterns rather than enforcing a uniform sparse structure, thereby reducing hardware overhead.
Solution Approach 2:
The patent introduces dynamics by making the sparsity threshold adjustable and adaptable during training. The threshold can be modified based on the specific layer characteristics, input data distribution, and training progress, allowing the hardware architecture to dynamically adapt to varying computational requirements rather than being fixed for all operations.
3Quantity of substance
If sparsification is applied to activation gradients, then memory demand is reduced, but training accuracy may decrease
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the sparsity threshold based on the specific neural network layer, input data characteristics, and training progress. This allows the system to maintain higher precision when needed while achieving greater sparsity when appropriate, thereby balancing memory demand reduction with training accuracy preservation.
Solution Approach 2:
The patent implements feedback by monitoring the impact of sparsification on gradient quality and adjusting the sparsity threshold accordingly. If sparsification begins to significantly degrade training accuracy, the threshold is adjusted to retain more gradient information, creating a closed-loop system that adapts to maintain accuracy while minimizing memory usage.
Data Source
AI summary
A processor-implemented neural network operation method includes: receiving a first activation gradient and a first threshold corresponding to a layer included in a neural network; sparsifying the first activation gradient based on the first threshold; determining a second activation gradient by performing a neural network operation based on the sparsified first activation gradient; determining a second threshold by updating the first threshold based on the second activation gradient; and performing a neural network operation based on the second activation gradient and the second threshold.