Sparse Weight Matrix Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing size and training time of neural network models necessitate improved systems and methods for efficient training, as conventional methods are inefficient due to redundant calculations and high memory usage.
Innovation Solution
The use of sparse matrix-matrix multiplication (spMM) and sampled dense-dense matrix multiplication (SDDMM) modules, along with transpose invariant sparse weight matrices, to compute activations and gradients in forward and backward passes, reducing redundant calculations and memory consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional dense matrix multiplication is used for neural network training, then computation is straightforward and easy to implement, but training time increases and memory usage increases due to redundant calculations with zero-value elements
Solution Approach 1:
The patent segments the weight matrix into sparse format by dividing it into non-zero elements and their corresponding indices. This segmentation allows the system to process only relevant computations while eliminating redundant operations with zero-value elements, thereby improving training speed without excessive complexity
Solution Approach 2:
The patent dynamically switches between sparse matrix operations for forward propagation and dense matrix operations for backward propagation based on computational requirements. This dynamic approach optimizes training efficiency by applying the appropriate computation mode at each stage, resolving the contradiction between speed and complexity
2Loss of time
If sparse weight matrices are used to eliminate redundant calculations, then training efficiency improves, but additional complexity is introduced in managing sparse data structures and transposes
Solution Approach 1:
The patent extracts only the non-zero elements and their indices from the weight matrix, storing them in compact sparse formats. This extraction eliminates the need to store and process zero-value elements, reducing training time while managing complexity through focused data representation
Solution Approach 2:
The patent changes the representation parameters of the weight matrix from dense format to sparse format with explicit index tracking. This parameter change enables efficient storage and computation by adapting the data structure to the actual distribution of non-zero elements, reducing training time without proportionally increasing complexity
3Reliability
If full dense matrix operations are performed during backward pass, then gradient computation is accurate and complete, but memory consumption increases significantly
Solution Approach 1:
The patent extracts and processes only the necessary gradient computations using sparse operations where applicable, rather than performing full dense matrix operations. This extraction maintains gradient accuracy for non-zero elements while significantly reducing memory consumption by excluding operations involving zero-value elements
Solution Approach 2:
The patent dynamically adapts the computation mode during backward propagation, using sparse operations for weight gradient computation when the weight matrix is sparse, and dense operations only where necessary. This dynamic approach maintains gradient accuracy while optimizing memory usage according to the actual data characteristics
Data Source
AI summary
Neural network (NN) model training techniques can include computing activations in a forward pass using a sparse weight matrix that is transpose invariant. The neural network (NN) model training techniques can further include computing activation gradients and weight gradients in a backward pass using the sparse weight matrix.