Neural Network Model Optimization via Layer Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional optimization methods for neural network models are insufficient in achieving performance, efficiency, and form optimization beyond minimizing the loss function, particularly in reducing model size and inference time.
Innovation Solution
A method and device for optimizing neural network models by adjusting the number of hidden layers through layer and block fusion, reducing activation functions, and fusing batch normalization with fully connected layers, while minimizing the number of hidden layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional optimization methods are used to minimize loss function, then model performance is improved, but model size and inference time are not reduced
Solution Approach 1:
The patent merges batch normalization layers with fully connected layers into unified fusion layers, and combines consecutive blocks into single blocks. This merging reduces the number of separate operations and layers, thereby reducing inference time while maintaining model performance through the preserved functional equivalence of the fused structures.
Solution Approach 2:
The patent extracts and removes redundant activation functions from between consecutive blocks, and eliminates unnecessary hidden layers. By taking out these redundant components, the model achieves faster inference time while the essential computational functionality is preserved through the remaining fused layers.
2Reliability
If conventional optimization methods are used, then loss function is minimized, but computational operations are not reduced
Solution Approach 1:
The patent combines multiple sequential operations (batch normalization followed by fully connected layer, or vice versa) into single fusion layers that perform both operations simultaneously. This merging reduces the total number of computational operations while maintaining the same mathematical transformation, thereby improving productivity.
Solution Approach 2:
The fusion layers created in the patent serve multiple functions simultaneously - they perform batch normalization, fully connected transformations, and activation functions within a single layer structure. This multi-functionality reduces the number of separate computational steps required, improving operational efficiency.
3Quantity of substance
If number of hidden layers is reduced, then model size is decreased, but computational capability may be compromised
Solution Approach 1:
The patent merges the functionality of multiple hidden layers into fewer fusion layers by combining batch normalization and fully connected operations. This merging reduces the number of hidden layers while the fused layers compensate for the reduction by performing multiple transformations simultaneously, thereby maintaining computational capability with a smaller model size.
Solution Approach 2:
The patent changes the structural parameters of the network by reducing the number of hidden layers and modifying the architecture through fusion. By adjusting these parameters and optimizing the fused layer configurations, the model achieves smaller size while preserving necessary computational capability through the enhanced fusion operations.
Data Source
AI summary
Disclosed is a method of optimizing a neural network model, performed by a neural network model optimization device, the method including fusing batch normalization and a fully connected layer included in a block included in the neural network model; removing an activation function included in one block between consecutive first block and second block; and when the activation function is absent in the preceding first block, fusing the first block and the succeeding second block.