Locality-Sensitive Hashing for CNN Channel Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for reducing computational cost in convolutional neural networks (CNNs) require retraining or fine-tuning, leading to increased energy and time consumption, and are dependent on the original training data, which can result in performance loss and data dependency issues.
Innovation Solution
A method using locality-sensitive hashing (LSH) for channel pruning in CNNs, allowing for a plug-and-play replacement of convolutional modules that reduces floating-point operations (FLOPs) without the need for retraining or fine-tuning, by dynamically reducing the number of channels in convolutional layers through hashing and merging redundant information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional pruning methods (e.g., filter pruning, channel pruning) are applied to reduce computational cost, then FLOPs are reduced, but retraining or fine-tuning is required which increases energy and time consumption
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing hash values for all possible filter channels during an offline phase. This preprocessing allows the runtime system to quickly identify and prune redundant channels through hash matching without requiring any retraining or fine-tuning, thus resolving the contradiction between computational efficiency and retraining time
Solution Approach 2:
The patent uses copying by creating hash representations (copies) of filter channel data that can be quickly compared and matched. Instead of working with the full high-dimensional filter data during pruning, the system uses compact hash copies to identify redundant channels, enabling fast pruning decisions without time-consuming retraining
2Device complexity
If filter channels are pruned to reduce model size and FLOPs, then computational cost decreases, but model accuracy may deteriorate without fine-tuning
Solution Approach 1:
The patent applies self-service by designing a pruning mechanism that is entirely self-contained and does not require external fine-tuning processes. The hash-based redundancy detection automatically identifies and prunes redundant channels while preserving necessary computational paths, allowing the model to maintain its accuracy without needing external optimization steps
Solution Approach 2:
The patent changes parameters by transforming filter channel data into hash representations, fundamentally altering the data format from high-dimensional floating-point values to compact hash codes. This parameter transformation enables efficient redundancy detection and pruning while maintaining the integrity of non-redundant channels, thus preserving model accuracy
3Use of energy by moving object
If pruning is applied to reduce FLOPs, then energy consumption decreases, but dependency on original training data increases
Solution Approach 1:
The patent extracts the essential redundancy information from the training data into hash representations during an offline phase. By extracting and storing only the necessary hash values rather than the complete training dataset, the system enables pruning without requiring access to the original training data during deployment, thus reducing data dependency while maintaining low energy consumption
Data Source
AI summary
A computer-implemented method for a computer-implemented method of efficiently calculating convolution operations. The method includes receiving a tensor of input data to be proceeded and at least one filter and initializing a locality-sensitive hashing function. Then, repeating the following steps for each patch in the tensor: Slicing the current receptive field into a series of matrices. Applying the locality-sensitive hashing to each of said matrices to determine a hash representation for each matrix. Merging the matrixes with essentially the same hash representation to a new matrix. Creating a reduced tensor by arrange the merged matrices in a series. Merging the filter coefficients in the same order as the matrices have been merged and convolving the merged on the reduced sub-tensor with the merged filter.