Sparsifying Neural Network Models via Group Lasso Regularization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep convolutional neural networks (CNNs) require significant memory and computational resources due to large amounts of data representing kernel weights, making them challenging to implement on resource-constrained devices like smartphones and tablets, and existing sparsification methods like random pruning result in poor data locality and high index overhead.
Innovation Solution
Applying a group lasso regularization constraint during training to identify and group kernel elements with common dimensions, allowing for contiguous storage and compression of zero weights, and using a bit string representation to bypass multiplication operations involving zero values, thereby reducing memory footprint and improving data locality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If random pruning is used to sparsify the neural network model, then the memory footprint is reduced, but data locality deteriorates and index overhead increases
Solution Approach 1:
The patent segments the kernel weights into multiple groups based on their dimensional characteristics. By organizing weights into structured groups rather than random removal, the patent achieves sparsity while maintaining data locality within each group, thereby reducing index overhead compared to random pruning approaches.
Solution Approach 2:
The patent applies different treatment to different parts of the kernel weights based on their local dimensional properties. By identifying and grouping kernel elements with common dimensions, the patent creates regions of high data locality for non-zero weights while maintaining overall model sparsity, thus improving memory efficiency without sacrificing computational performance.
2Reliability
If the neural network model uses full precision kernel weights, then the accuracy is maintained, but the memory and computational resources required increase significantly
Solution Approach 1:
The patent changes the parameter representation of kernel weights by organizing them into grouped structures with shared dimensional properties. This parameter reorganization enables compact storage of non-zero weights while maintaining their precise values, thus preserving model accuracy while reducing memory footprint and computational resource requirements through efficient data layout.
3Quantity of substance
If the neural network model is sparsified to reduce resource requirements, then the memory footprint is reduced, but the computational efficiency may deteriorate
Solution Approach 1:
By segmenting kernel weights into structured groups based on dimensional commonality, the patent enables efficient computational processing of sparse weights. The segmented structure allows for optimized computation patterns that maintain computational efficiency while achieving significant memory footprint reduction through systematic sparsification.
Solution Approach 2:
The patent improves computational efficiency by creating local regions of high data locality within grouped kernel weights. This local organization allows processing units to efficiently access and compute with non-zero weights in contiguous memory regions, thereby maintaining computational throughput while benefiting from reduced memory access overhead compared to random sparsification methods.
Data Source
AI summary
A technique includes modifying a neural network model to sparsify the model. The model includes a plurality of kernel element weights, which are parameterized according to a plurality of dimensions. Modifying the model includes, in a given iteration of the plurality of iterations, training the model based on a structure regularization in which kernel element weights that share a dimension in common are removed as a group to create corresponding zero kernel elements in the model; and compressing the model to exclude zero kernel element weights from the model to prepare the model to be trained in another iteration.