Neural Network Layer Folding for Lower-Power Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As neural networks increase in depth and size, they demand significant computational resources, leading to increased power consumption and latency, which is particularly problematic in devices with limited processing and storage capacities, necessitating efficient reduction techniques without compromising accuracy.
Innovation Solution
The method involves replacing non-linear activation units with an identity function and collapsing linear blocks to form a single block, iteratively adjusting a linearity parameter to obtain an approximately affine activation function, and combining it with the affine function to reduce the number of layers in the neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the depth and size of neural network increase to improve accuracy, then the processing performance improves, but power consumption and computational complexity increase
Solution Approach 1:
The patent merges multiple linear layers into a single folded layer by replacing non-linear activation functions with identity functions. This combining operation reduces the total number of layers and operations while maintaining the network's representational capacity, thereby decreasing power consumption without significantly compromising accuracy.
Solution Approach 2:
The patent introduces a learnable parameter α that controls the transformation of activation functions. By optimizing this parameter during training, the network learns when and where to replace non-linear activations with identity functions, enabling dynamic parameter adjustment that balances accuracy and computational efficiency.
2Measurement precision
If the depth and size of neural network increase to improve accuracy, then the processing performance improves, but computational complexity increases
Solution Approach 1:
The patent combines multiple sequential linear layers into a single folded layer by eliminating non-linear activation functions. This merging reduces the total number of operations and layers in the network, directly decreasing computational complexity while preserving the essential transformation capabilities through optimized parameter learning.
3Use of energy by moving object
If neural network reduction techniques are applied to decrease layers and operations, then power consumption and computational complexity reduce, but accuracy may be compromised
Solution Approach 1:
The patent uses learnable parameters to control the folding process, allowing the network to adaptively determine which activations to replace. This parameter-based approach enables the network to maintain accuracy by learning optimal folding configurations during training, rather than applying fixed reduction rules that might compromise performance.
Solution Approach 2:
The patent incorporates loss functions that provide feedback during training to guide the folding process. This feedback mechanism allows the network to learn when folding operations are beneficial and when to preserve non-linear activations, ensuring accuracy is maintained while achieving power consumption reductions.
4Device complexity
If neural network reduction techniques are applied to decrease layers and operations, then computational complexity reduces, but accuracy may be compromised
Solution Approach 1:
The patent introduces learnable parameters that control the folding operation, allowing the network to dynamically adjust its structure based on the data. This parameter-based folding maintains accuracy by learning optimal reduction configurations during training, rather than applying static compression that would compromise performance.
Data Source
AI summary
The present disclosure describes neural network reduction techniques for decreasing the number of neurons or layers in a neural network. Embodiments of the method, apparatus, non-transitory computer readable medium, and system are configured to receive a trained neural network and replace certain non-linear activation units with an identity function. Next, linear blocks may then be folded to form a single block in places where the non-linear activation units were replaced by an identity function. Such techniques may reduce the number of layers in the neural network, which may optimize power and computation efficiency of the neural network architecture (e.g., without unduly influencing the accuracy of the network model).