CNN Compression Using Filter Pruning and Tensor Decomposition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing CNNs are too large to fit in systems with limited computing resources, and existing compression methods like Lottery Ticket Pruning (LTP) are time-consuming and require significant computing resources, while low-rank tensor decomposition fails to reduce size sufficiently without losing accuracy.
Innovation Solution
A method combining filter pruning and tensor decomposition, involving a one-shot pruning step followed by tensor decomposition to reduce CNN size, with a fine-tuning process to recover accuracy, requiring fewer computational resources and time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of moving object
If filter pruning is applied to reduce CNN size, then model size is reduced, but accuracy is lost
Solution Approach 1:
The patent applies preliminary action by performing filter pruning first to reduce model size, then followed by tensor decomposition and fine-tuning to recover accuracy. The pruning step removes less important filters before the network is retrained and refined through subsequent decomposition and fine-tuning processes.
Solution Approach 2:
The patent changes parameters through tensor decomposition which transforms the pruned convolutional layers into a more compact representation. This parameter transformation allows the model to maintain its functionality with fewer parameters, and fine-tuning further adjusts these parameters to recover accuracy after pruning.
2Volume of moving object
If low-rank tensor decomposition is used to reduce CNN size, then model size is reduced, but accuracy loss is significant
Solution Approach 1:
The patent applies preliminary action by performing filter pruning first to reduce model size, then followed by tensor decomposition and fine-tuning to recover accuracy. The pruning step removes less important filters before the network is retrained and refined through subsequent decomposition and fine-tuning processes.
Solution Approach 2:
The patent uses a composite approach by combining filter pruning with tensor decomposition and fine-tuning. This composite method integrates multiple techniques where pruning provides initial compression, tensor decomposition adds structural efficiency, and fine-tuning restores accuracy, achieving better results than any single method alone.
3Volume of moving object
If Lottery Ticket Pruning (LTP) is applied to compress CNN, then model size is reduced, but computational resources and time are excessive
Solution Approach 1:
The patent applies preliminary action by performing filter pruning first to reduce model size, then followed by tensor decomposition and fine-tuning to recover accuracy. The pruning step removes less important filters before the network is retrained and refined through subsequent decomposition and fine-tuning processes.
Solution Approach 2:
The patent changes parameters through tensor decomposition which transforms the pruned convolutional layers into a more compact representation. This parameter transformation allows the model to maintain its functionality with fewer parameters, and fine-tuning further adjusts these parameters to recover accuracy after pruning.
Data Source
AI summary
An apparatus is provided to compress CNN models using a combination of filter pruning and tensor decomposition. For example, the apparatus accesses a trained CNN that includes convolutional tensors. The apparatus prunes the filters of a convolutional tensor to generate a sparse tensor. Further, the apparatus decomposes the sparse tensor to generate a low-rank approximation of the sparce tensor. The low-rank approximation of the sparse tensor includes a core tensor and principal tensors. The apparatus generates a convolutional flow that includes the core tensor and convolutional operations generated based on the principal tensors. The apparatus may replace some or all the convolutional tensors in the trained CNN with the corresponding convolutional flows. The apparatus may fine-tune the updated CNN by re-training the updated CNN. The number of epochs for re-training the updated CNN may be smaller than the number of epochs for training the CNN.