Perturbation Layers for Low-Complexity Neural Feature Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional layers in deep neural networks are computationally intensive and remain a bottleneck, limiting the efficiency of image recognition tasks.
Innovation Solution
Replace convolutional layers with perturbation layers that use random, fixed noise masks and weighted non-linear activations to generate feature maps, eliminating the need for traditional convolutional operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If convolutional layers are used for image recognition, then feature extraction capability is improved, but computational complexity increases
Solution Approach 1:
The patent segments the convolutional layer into two separate components: a perturbation layer that applies random noise masks to divide the input space, and a linear layer that performs weighted combinations. This segmentation eliminates the computationally intensive convolution operation while preserving feature extraction capability through the combination of perturbed inputs.
Solution Approach 2:
The patent substitutes the mechanical convolution operation with a different computational mechanism: adding random noise masks followed by linear weighted combinations. This replacement eliminates the need for sliding window convolutions and their associated computational complexity while maintaining the ability to extract meaningful features from images.
2Productivity
If convolutional layers with small receptive fields are used, then computational efficiency is improved, but model performance deteriorates
Solution Approach 1:
The patent changes the dimensionality of the computational approach by moving from spatial convolutions with small receptive fields to a different dimensional space where random noise masks are added and then combined linearly. This dimensional change allows the model to achieve both computational efficiency and high performance by operating in a transformed computational space rather than being constrained to small spatial windows.
3Use of energy by moving object
If convolutional layers are replaced with perturbation layers, then computational requirements are reduced, but feature extraction capability may deteriorate
Solution Approach 1:
The patent applies preliminary action by pre-defining random noise masks that are added to the input before the main computation. These perturbation masks are fixed and do not require learning, yet they effectively divide the input space and enable the subsequent linear layer to extract features efficiently. This preliminary perturbation step reduces computational requirements while preserving feature extraction capability.
Solution Approach 2:
The patent changes the parameters of the computational system by replacing learnable convolutional kernels with fixed random noise masks and learnable linear weights. This parameter change transforms the problem from learning complex convolutional filters to learning simple linear combinations of perturbed inputs, reducing computational requirements while maintaining feature extraction effectiveness.
Data Source
AI summary
Perturbative neural networks are comprised of one or more modules, called perturbation layers which can be used as an alternative to a convolutional layer. The perturbation layer does away with convolution in the traditional sense and instead computes its response as a weighted linear combination of non-linearly activated additive noise perturbed inputs.