Partial Convolution Neural Network for Low Latency Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks fail to achieve genuinely low latency despite reducing floating-point operations (FLOPs), often due to inefficient memory access and computational redundancy, leading to a discrepancy between FLOPs and actual speed.
Innovation Solution
The implementation of a partial convolution (PConv) operator that selectively convolves only a portion of input channels, leveraging redundancy in feature maps, combined with pointwise convolution (PWConv) layers to reduce memory access and computational complexity, resulting in higher FLOPS and lower latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If depthwise convolution and group convolution are used to reduce FLOPs, then computational complexity is reduced, but memory access increases
Solution Approach 1:
The patent segments the convolution operation into two distinct parts: PConv that processes only salient channels with spatial filters, and PWConv that processes all channels with 1x1 filters. This segmentation allows the network to avoid unnecessary computations on non-salient channels while maintaining complete channel information flow, thereby reducing both FLOPs and memory access requirements compared to conventional depthwise or group convolution approaches.
2Device complexity
If network decomposition and sparsification are applied to achieve low FLOPs, then computational operations are reduced, but computation efficiency decreases due to fragmented computation
Solution Approach 1:
The patent maintains continuity of useful action by ensuring that the PWConv layer processes all channels continuously after the PConv layer, preventing fragmentation of the computation pipeline. The two convolution operations are seamlessly connected in sequence, allowing efficient data flow and computation without the need for complex reorganization or supplementary data manipulations that would break the computational continuity.
3Adaptability or versatility
If supplementary data manipulations such as concatenation, shuffling, and pooling are added, then feature processing capability is enhanced, but runtime increases significantly
Solution Approach 1:
The patent applies local quality by making the convolution operation adaptive to different channels through the PConv layer, which identifies and processes only salient channels with spatial filters. This localized processing approach enhances feature extraction capability for important channels while avoiding unnecessary operations on non-salient channels, thereby improving feature processing efficiency without requiring supplementary data manipulations that would increase runtime.
4Reliability
If advanced normalization and activation layers are used, then model performance is improved, but computational speed is limited
Solution Approach 1:
The patent extracts and removes unnecessary computational overhead by eliminating advanced normalization and activation layers that do not contribute significantly to performance on mobile devices. The simplified architecture relies on the efficient PConv and PWConv operations to deliver both high performance and fast computational speed, taking out the bottlenecking components that slow down inference on resource-constrained devices.
Data Source
AI summary
A device for employing an efficient neural network architecture through partial convolution is provided, including a fast network module, a data input module, and an outcome module. A fast neural network including multiple fast neural network blocks with at least one PConv layer and at least two PWConv layers are integrated in the fast network module. The data input module is responsible for loading and providing input data to the fast network module. The PConv layer is applied for partial convolution of the input data with achieving standard convolution operations on partial channels while preserving other channels unaffected and selectively convolves only a portion of input channels by leveraging redundant information in feature maps. The two PWConv layers following the PConv layer are configured to transform and integrate features. The outcome module is configured to receive results generated by the fast network module.