CNN Bottleneck Blocks for Low-Cost Mobile Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks (CNNs) have become computationally expensive and memory-intensive, making them challenging for mobile and embedded computing environments, and existing compression techniques either result in irregular networks or overly complex models.
Innovation Solution
Implementing neural networks with convolutional blocks that include linear bottleneck layers and inverted residual blocks, utilizing depthwise separable convolutions and residual shortcut connections to reduce computational and memory requirements while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If convolutional neural networks increase in size and complexity to achieve higher accuracy, then accuracy is improved, but computational cost and memory requirements increase
Solution Approach 1:
The patent segments the convolutional operation into two separate operations: depthwise convolution (applying one filter per input channel) and pointwise convolution (1x1 convolutions to combine channels). This segmentation reduces computational complexity from O(k^3) to O(k^2) where k is the number of channels, achieving efficiency while maintaining accuracy
Solution Approach 2:
The patent inverts the traditional residual connection order by applying the nonlinear activation function (ReLU) before the convolutional operation rather than after. This inversion allows the network to learn residual functions in a more efficient manner, reducing computational cost while preserving accuracy
2Measurement precision
If convolutional neural networks increase in size and complexity to achieve higher accuracy, then accuracy is improved, but memory requirements increase
Solution Approach 1:
By segmenting the convolution operation into depthwise and pointwise components, the patent reduces the number of parameters that need to be stored in memory. The depthwise convolution uses fewer filters, and the pointwise convolution efficiently combines channels, collectively reducing memory requirements while maintaining representational capacity for high accuracy
3Use of energy by moving object
If network pruning is applied to reduce computational cost, then computational cost decreases, but the network becomes irregular and requires extra efforts to represent
Solution Approach 1:
The patent changes the parameters of the convolutional operation by using depthwise separable convolutions with specific kernel sizes (e.g., 3x3 depthwise + 1x1 pointwise) and controlling the expansion ratio. This parameterization maintains regular network structure while achieving computational efficiency comparable to or better than pruned networks
Data Source
AI summary
The present disclosure provides directed to new, more efficient neural network architectures. As one example, in some implementations, the neural network architectures of the present disclosure can include a linear bottleneck layer positioned structurally prior to and/or after one or more convolutional layers, such as, for example, one or more depthwise separable convolutional layers. As another example, in some implementations, the neural network architectures of the present disclosure can include one or more inverted residual blocks where the input and output of the inverted residual block are thin bottleneck layers, while an intermediate layer is an expanded representation. For example, the expanded representation can include one or more convolutional layers, such as, for example, one or more depthwise separable convolutional layers. A residual shortcut connection can exist between the thin bottleneck layers that play a role of an input and output of the inverted residual block.