Deep Neural Network Compression via Filter Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) require significant computational power and storage, making them challenging for mobile devices and vehicles, and existing pruning methods often compromise generalizability or are inefficient.
Innovation Solution
A method involving filter similarity pruning using subspace clustering and dimension reduction to compress DNNs, preserving structural integrity and reducing storage without sacrificing accuracy, through iterative processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are trained to achieve high accuracy on large datasets, then model accuracy is improved, but model size and computational resources required increase
Solution Approach 1:
The patent segments the neural network into a teacher network and a student network, where the teacher network performs the heavy computational task of training on large datasets while the student network stores compressed representations. This segmentation allows high accuracy to be achieved without requiring the deployed model to be large, as the student network only needs to store compressed features rather than all training data.
Solution Approach 2:
The patent extracts essential information from the training data and represents it in a compressed form within the student network. By taking out only the necessary features and representations while discarding redundant information, the model achieves high accuracy with reduced model size and computational requirements.
2Measurement precision
If neural networks are trained on large datasets, then model accuracy improves, but training time and computational resources increase
Solution Approach 1:
The patent performs preliminary action by pre-training the teacher network on large datasets before deploying the student network. The heavy computational work of training is completed in advance, and the student network only needs to load and use the pre-computed compressed representations, significantly reducing the training time required for deployment.
Solution Approach 2:
The patent creates a compressed copy of the essential learning patterns from the training data and stores them in the student network. This copying approach allows the model to capture the essential information from large datasets without requiring to process all data during deployment, reducing computational time and resources.
3Device complexity
If model parameters are reduced for deployment, then computational resources decrease, but model accuracy may deteriorate
Solution Approach 1:
The patent changes the parameter representation from storing all training data and model weights to storing compressed feature representations. This parameter change allows the model to maintain high accuracy with fewer parameters, as the compressed representations capture the essential learning patterns without requiring the full training data to be stored or processed.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system and a method are provided for compressing a deep neural network ("DNN"). In some examples, the DNN is trained, where the DNN has at least one layer having multiple filters. Clustering of the filters of at least one layer is performed. Dimension reduction can be applied as well to the filters to reduce the channel dimensionality of the at least one layer. The dimensionally reduced DNN can then be retrained. Once retrained, the compressed DNN can be stored in a storage device.