Neural Network Layer Reordering for Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial neural networks require significant memory resources and processing cycles for execution, especially when partially binary quantized, which affects their efficiency and accuracy.
Innovation Solution
The method involves reordering the layers of a trained neural network by displacing the batch normalization layer after the convolution layer, allowing the batch normalization layer to be merged with the convolution layer and converting the pooling layer into a binary form, thereby reducing memory occupation and processing cycles without compromising accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the neural network is quantized to binary format to reduce memory requirements and accelerate execution, then memory usage and processing time are reduced, but the accuracy and flexibility of the network may deteriorate
Solution Approach 1:
The patent applies parameter changes by transforming the batch normalization layer into a binary format that matches the quantized convolution and pooling layers. This involves changing the precision parameters of the batch normalization operations to binary, allowing the entire network to execute in binary while maintaining accuracy through mathematical equivalence transformations.
Solution Approach 2:
The patent inverts the traditional approach by instead of using high-precision batch normalization with binary convolution/pooling, it transforms the batch normalization itself to binary operations. This inversion allows all layers to operate at binary precision, maximizing speed and memory efficiency while preserving accuracy through the mathematical equivalence of the transformed operations.
2Measurement precision
If the batch normalization layer is kept after the pooling layer to maintain training accuracy, then accuracy is preserved, but execution time and memory occupation increase
Solution Approach 1:
The patent merges the batch normalization layer with the binary convolution and pooling layers by transforming batch normalization into equivalent binary operations that can be executed simultaneously with the surrounding binary layers. This merging eliminates the need for separate high-precision batch normalization execution, reducing total execution time and memory occupation while maintaining accuracy through the mathematical equivalence of the combined operations.
3Measurement precision
If the neural network layers are ordered with batch normalization after pooling for accurate training, then training accuracy is maintained, but the number of processing cycles increases
Solution Approach 1:
The patent changes the operational parameters of the batch normalization layer from high-precision floating-point operations to binary operations. This parameter change allows the batch normalization to be executed in the same binary domain as the convolution and pooling layers, reducing the number of processing cycles required while maintaining training accuracy through the mathematical equivalence of the transformed operations.
Data Source
AI summary
According to one aspect, there is proposed a method for transforming a trained artificial neural network including a binary convolution layer followed by a pooling layer then a batch normalization layer, the method includes obtaining the trained artificial neural network and transforming the trained artificial neural network such that the order of the layers of the trained artificial neural network is modified by displacing the batch normalization layer after the convolution layer.