Separable Convolution Network Layers for Lower Memory and Compute
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network systems face high memory usage and processing time due to the large amount of calculation required for convolution operations, particularly in convolutional neural networks.
Innovation Solution
Implement a neural network system with a separable convolution subnetwork that includes a sequence of separable convolutional neural network (SCNN) layers, which apply a first grouped convolution and a pointwise convolution in sequence to reduce the calculation burden.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If standard convolution operations are used in neural networks, then learning effectiveness is maintained, but memory usage and processing time become extremely large
Solution Approach 1:
The patent divides the standard convolution operation into two separate operations: depthwise convolution (applying one filter per input channel) and pointwise convolution (1x1 convolutions to combine channels). This segmentation reduces the computational complexity from O(k^2 * c_in * c_out) to O(k^2 * c_in + c_in * c_out), where k is the kernel size, significantly reducing processing time while maintaining learning effectiveness through the combination of both operations.
2Reliability
If standard convolution operations are used in neural networks, then learning effectiveness is maintained, but memory usage becomes extremely large
Solution Approach 1:
By segmenting the convolution into depthwise and pointwise operations, the patent reduces the number of parameters that need to be stored. The depthwise convolution stores c_in * k^2 parameters instead of k^2 * c_in * c_out, and the pointwise convolution stores c_in * c_out parameters. This segmentation dramatically reduces memory usage for storing convolution kernels and intermediate activations.
3Productivity
If channels are divided into groups for convolution, then calculation requirements are reduced, but connectivity sparsity increases
Solution Approach 1:
The patent applies segmentation by dividing channels into groups for depthwise convolution, where each group is processed independently. This reduces calculation requirements by a factor of g (number of groups). The subsequent pointwise convolution restores full connectivity by combining all groups, thus balancing computational efficiency with connectivity needs.
Solution Approach 2:
The patent transitions from spatial convolution to channel-wise convolution in the depthwise step, operating in a different dimension (channel dimension instead of spatial dimension). This dimensional change enables the grouping strategy to reduce computation while the pointwise convolution in the original spatial dimension restores full interactions.
Data Source
AI summary
A neural network system includes a separable convolution subnetwork. The separable convolution subnetwork includes a plurality of separable convolutional neural network (SCNN) layers arranged in a stack manner in sequence. Each of the plurality of SCNN layers applies a first grouped convolution to an input to the SCNN layer. An input to the first grouped convolution includes a plurality of channels, and the first grouped convolution is a spatial convolution which divides channels of an input to the first grouped convolution into groups in a channel-wise manner, convolves the grouped channels, and couples the convolved channels to generate an output.