Decomposing 2-D Convolution Operations in Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training methods face challenges in efficiently reducing the computational complexity of convolution operations, particularly in deep convolutional networks, due to the high computational intensity of 2-D convolution operations.
Innovation Solution
The method involves decomposing 2-D convolution operations into a composition of 1-D convolution operations by expressing the filter matrix as a linear combination of unit rank matrices using Singular Value Decomposition (SVD), allowing for a low-rank approximation of filter weight matrices and applying a decomposed convolution approach.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 2-D convolution operations are used in neural networks, then filtering and feature extraction capability is improved, but computational complexity increases
Solution Approach 1:
The patent segments the 2-D convolution operation into two separate 1-D convolution operations: first convolving along rows, then convolving along columns. This segmentation reduces computational complexity from O(M×N×K×L) to O(M×N×(K+L)) while maintaining the same filtering capability, as the separable filter can be decomposed into horizontal and vertical components that are applied sequentially
Solution Approach 2:
The patent transforms the 2-D convolution problem into a composition of 1-D convolutions by introducing a temporal or sequential dimension. Instead of applying a 2-D filter simultaneously across both dimensions, the operation is unfolded into two sequential 1-D passes, effectively changing the dimensionality of the operation from simultaneous 2-D processing to sequential 1-D processing
2Measurement precision
If standard convolution operations are applied, then neural network accuracy is improved, but training time increases
Solution Approach 1:
By segmenting the convolution operation into two 1-D passes, the patent reduces the number of multiplications and additions required during training. The computational burden is distributed across two simpler operations rather than one complex operation, maintaining accuracy while reducing training time
3Measurement precision
If 2-D convolution operations are used, then feature representation quality is improved, but resource utilization deteriorates
Solution Approach 1:
The patent improves resource utilization by segmenting the 2-D convolution into two 1-D convolutions. This allows for more efficient use of computational resources, as 1-D convolutions require fewer arithmetic operations and can be implemented with lower memory bandwidth requirements, thereby improving overall system productivity while maintaining feature representation quality
Data Source
AI summary
A method of training a neural network includes encouraging one or more filters in the neural network to have a low rank.