3D CNN Kernel Compression via Orthogonal Time-Dimension Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for compressing three-dimensional convolutional neural networks (3D CNNs) are ineffective in reducing their parameter quantity and calculation amount, making them difficult to implement on edge devices and requiring significant computational resources.
Innovation Solution
Perform orthogonal transform on the time dimension of 3D CNN kernels and prune the sparse time dimension to reduce the parameter and calculation load, using learnable orthogonal matrices optimized through target functions to maintain efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If three-dimensional convolution is used to process video and three-dimensional images, then processing capability for spatiotemporal tasks is improved, but parameter quantity and calculation amount increase significantly
Solution Approach 1:
The patent segments the three-dimensional convolution operation by introducing an orthogonal transform that decomposes the 3D convolution kernel into multiple two-dimensional convolution kernels. This segmentation allows the complex 3D convolution to be broken down into simpler 2D operations, reducing the overall parameter quantity and calculation amount while maintaining processing capability for spatiotemporal tasks.
Solution Approach 2:
The patent applies dimensionality change by introducing an orthogonal transform in the channel dimension, converting the three-dimensional convolution operation into a combination of two-dimensional convolution operations. This dimensional transformation reduces the computational complexity from O(d×d×t×c×N) to O(d×d×(t×c)×N), effectively lowering the parameter quantity while preserving the ability to process video and three-dimensional image data.
2Reliability
If three-dimensional convolution is used for video processing, then mutual information in time dimension is captured, but calculation amount increases
Solution Approach 1:
The patent segments the time dimension processing by decomposing the 3D convolution kernel into multiple 2D kernels through orthogonal transform. Each 2D kernel processes temporal information independently, allowing the model to capture mutual information across time frames while reducing the computational burden of processing all time dimensions simultaneously in a single 3D operation.
Solution Approach 2:
The patent replaces the computationally intensive mechanical three-dimensional convolution operation with a transformed system using orthogonal matrices. This substitution converts the direct 3D convolution into a sequence of 2D convolutions followed by a linear transformation, significantly reducing calculation amount while preserving the ability to capture temporal mutual information in video data.
3Quantity of substance
If current compression methods are applied to three-dimensional convolutional neural networks, then some parameter reduction is achieved, but calculation amount cannot be effectively reduced
Solution Approach 1:
The patent addresses this contradiction by applying orthogonal transform in the channel dimension, which fundamentally changes the computational structure. This dimensional transformation converts 3D convolution into 2D convolution operations, achieving effective reduction in both parameter quantity and calculation amount, unlike conventional compression methods that only prune filters without changing the operational dimensionality.
Solution Approach 2:
The patent changes the parameters of the convolution operation by introducing an orthogonal transform matrix that reparameterizes the 3D convolution kernel. This parameter transformation allows the model to maintain its representational capacity while using fewer effective parameters and reducing calculation amount, as the transformed parameters enable more efficient computation through 2D operations.
4Productivity
If three-dimensional convolutional neural network is deployed on edge devices, then video processing capability is provided, but computational resources are insufficient
Solution Approach 1:
The patent segments the computationally heavy three-dimensional convolution into multiple lighter two-dimensional convolution operations through orthogonal transform. This segmentation makes the model suitable for edge devices by breaking down the complex 3D operation into manageable 2D steps that can be executed with limited computational resources while still providing video processing capability.
Solution Approach 2:
The patent enables deployment on edge devices by transforming the three-dimensional convolution operation into two-dimensional operations through orthogonal transform. This dimensionality change reduces the computational complexity from O(d×d×t×c×N) to O(d×d×(t×c)×N), making the model feasible for resource-constrained edge devices while maintaining video processing functionality.
Data Source
Figure 1~3
Figure 4~5
Figure 6
AI summary
A neural network compression method, a data processing method, and a related apparatus are provided, and relate to the field of artificial intelligence, and in particular, to the field of computer vision. The method includes: performing the following operations on a first three-dimensional convolution kernel included in a to-be-compressed three-dimensional convolutional neural network: performing orthogonal transform on the first three-dimensional convolution kernel in a time dimension of the first three-dimensional convolution kernel (S310); and pruning the time dimension of the first three-dimensional convolution kernel based on sparsity of the orthogonally transformed first three-dimensional convolution kernel in the time dimension, to obtain a compressed three-dimensional convolutional neural network (S320). In the foregoing method, orthogonal transform is performed on the three-dimensional convolution kernel in the time dimension of the three-dimensional convolution kernel of the to-be-compressed three-dimensional convolutional neural network, and the sparse time dimension of the orthogonally transformed three-dimensional convolution kernel is pruned. In this way, a parameter quantity and a calculation amount of a three-dimensional convolutional neural network can be further reduced.