Dynamic Triplet Convolution for CNN Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing dynamic convolution techniques for convolutional neural networks (CNNs) suffer from increased memory costs and performance limitations, especially when applied to larger CNNs, due to the reliance on linear combinations of multiple static convolutional filters.
Innovation Solution
The implementation of dynamic triplet convolution, which transforms a static multidimensional convolutional filter into a dynamic one by using multi-scale attention mechanisms to learn scalar kernels that dynamically scale the filter's dimensions based on input feature maps, thereby avoiding the need for additional static filters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If linear combinations of multiple static convolutional filters are used to implement dynamic convolution, then adaptability of the CNN is improved, but memory cost and device complexity increase
Solution Approach 1:
The patent segments the convolutional filter into multiple learnable filterlets along the output channel dimension. Instead of using linear combinations of multiple complete static filters, each filter is divided into smaller filterlets that can be independently selected and combined, reducing the overall memory requirement while maintaining adaptability.
Solution Approach 2:
The patent introduces a new dimension (the T dimension) to organize filterlets, transforming the traditional 4D filter structure into a 5D structure with an additional filterlet index dimension. This dimensional transformation enables more efficient memory organization and access patterns, reducing memory cost while preserving the dynamic adaptation capability.
2Adaptability or versatility
If linear combinations of multiple static convolutional filters are used to implement dynamic convolution, then adaptability of the CNN is improved, but device complexity increases
Solution Approach 1:
By segmenting filters into filterlets, the patent simplifies the dynamic filter construction process. Instead of managing multiple complete static filters and their linear combinations, the system only needs to manage and combine small filterlets, reducing device complexity while maintaining adaptability.
Solution Approach 2:
The patent applies local quality by allowing different regions (filterlets) of the filter to have different properties and be selectively activated. This localized approach enables adaptability without requiring complex global filter management, simplifying the overall device structure.
3Reliability
If multiple static convolutional filters are trained and stored for dynamic combination, then performance of the CNN is improved, but storage requirements increase
Solution Approach 1:
The patent segments each static filter into multiple smaller filterlets along the output channel dimension. This segmentation allows the system to store and selectively combine only the necessary filterlets for each inference task, reducing total storage requirements while maintaining performance through adaptive filter construction.
Solution Approach 2:
Instead of training and storing multiple complete static filters, the patent trains and stores multiple filterlets that can be partially combined to form dynamic filters. This partial action approach reduces storage requirements while still achieving the desired performance through selective combination of filterlets.
Data Source
AI summary
Methods, apparatus, systems and articles of manufacture (e.g., physical storage media) to implement dynamic triplet convolution for convolutional neural networks are disclosed. An example apparatus disclosed herein for a convolutional neural network is to calculate one or more scalar kernels based on an input feature map applied to a layer of the convolutional neural network, ones of the one or more scalar kernels corresponding to respective dimensions of a static multidimensional convolutional filter associated with the layer of the convolutional neural network. The disclosed example apparatus is also to scale elements of the static multidimensional convolutional filter along a first one of the dimensions based on a first one of the one or more scalar kernels corresponding to the first one of the dimensions to determine a dynamic multidimensional convolutional filter associated with the layer of the convolutional neural network.